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Abstract 

We propose a relative trace formula approach and state the corresponding fundamental lemma 
toward the global restriction problem involving Bessel or Fourier-Jacobi periods of unitary groups 
U„ X Um, extending the work of Jacquet-Rallis for m = n — 1 (which is a Bessel period). In particular, 
when m = 0, we recover a relative trace formula proposed by Flicker concerning Kloosterman/Fourier 
integrals on quasi-split unitary groups. As evidence for our approach, we prove the fundamental 
lemma for Un x Un in positive characteristics. 
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1 Introduction 



Recently, Jacquet and Rallis [JR] propose a new approach to the Gan-Gross-Prasad conjecture for unitary 
groups U„ X U„_i relating certain periods with special L- values (cf. [GGP]). It is based on a relative trace 
formula. In this paper, we extend this approach to all kinds of pairs U„ x Um where < m < n. If ri — m 
is odd, the related period is a Besscl period. If n — to is even, the related period is a Fourier- Jacobi period. 

Notations. We denote by the unit circle in the complex line C. For a number field k, we denote 
by Mfc the set of places of k. 

We denote by M^^s the afhne group of r x s matrices and = Mr^r- We denote by 1^ the identity 
matrix of rank r. 

We denote by IS*] the cardinality of a finite set S and by It the characteristic function of any set T. 
For a locally compact abelian topological group or a vector space X, we denote for its dual. For 
a smooth representation tt, we denote tt for its contragredient representation. 

Acknowledgements. The author would like to thank Shou-Wu Zhang for proposing this question. 
He also thanks Herve Jacquet and Shou-Wu Zhang for stimulating discussions and their interest in this 
work. 



1.1 Periods and special values of L- functions 

Let us consider a quadratic extension of number fields k/k' . Let r be the nontrivial element in Gal(fc/fc') 
and 7] the corresponding quadratic character of the idele group A'^ of k' . Let be a (non-degenerated) 
hermitian space over k (with respect to r) of dimension n with the hermitian form (— , — ) and W C V a 
subspace of dimension to such that the restricted hermitian form ( — , —)\w is non-degenerate. 

Let Un = V{V) and Um = U(iy) be the unitary groups. We identify U„i as a subgroup of U„ fixing all 
elements in the orthogonal complement C F of W. We define a unipotent subgroup U' = U[r ,„ C U„ 
(resp. U' = U[r „i+i) normalized and hence acted through conjugation by Um when rt — to is even (resp. 
odd). We define H' = U' » U„i which is a non-reductive group viewed as a subgroup of U„ x Um via the 
embedding into the first factor and the projection onto the second factor (see Section 4.1 and 5.1 for the 
precise definitions). Let tt (resp. a) be an irreducible tempered representation of U„(A') (resp. Um(A')) 
which occurs with multiplicity one in the space of cusp forms ^o(U„) (resp. Ao{lJm))- We denote by A-n 
(resp. Aa) the unique irreducible tt (resp. cr)-isotypic subspace in ^o(U„) (resp. .4o(Um))- 

First, we consider the case of Bessel periods where we require that n — to = 2r -I- 1 is odd. We have a 
generic character i/' of C/'(fc')\[/'(A') and extend it to a character of H'{k')\H'{A') trivially on U„i(A'). 
For (^7r G An and ipa- G Aa, we define 

H'(k')\H'(K') 

to be a Bessel period oi n ® a. It is conjectured that there is a nonzero Bessel period of representations 
in the Vogan i-packet of tt (g) cr if and only if the central special L-value X(i, BC(7r) x BC(cr)) ^ 0, where 
BC stands for the standard base change and the L-function is the one defined by the Rankin-Sclbcrg 
convolution on general linear groups (cf. [JPSS83]). 

Second, we consider the case of Fourier- Jacobi periods where we require that n — m = 2r is even. We 
have an automorphic (essentially Weil) representation v'^, ^ of H'{A') by choosing a nontrivial character 
ip' : k'\A' — > and a character /i : fc^\A^ — > such that /i^'x = realizing on certain space of 
Bruhat-Schwartz functions S. For ip^^ G A^r, fa G Aa and G 5, we define 

Ji7"^'-((^,,^,;0) = J f^®f,{h')0{h',(l>)dh' 

H'{k')\H'{A') 

to be a Fourier- Jacobi period of tt (g) cr (with respect to /i), where 0{h',(p) is a certain thcta series on 
iJ'(A') attached to (f>. It is conjectured that there is a nonzero Fourier- Jacobi period of representations 
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in the Vogan L-packct of tt ® cr if and only if the central special L- value L(i, BC(7r) x BC(cr) /i ^) ^ 0. 

There has been significant progress toward this conjecture in a series of papers of Ginzburg-Jiang- 
Rallis: [GJR09] for unitary groups, [GJR04] for symplectic groups and [GJR05] for orthogonal groups. 
In all these cases, they prove one direction: nontrivial Bessel or Fourier- Jacobi periods imply the non- 
vanishing of corresponding central L-values. For the other direction, they also obtain some conditional 
results. Their approach is to study the residue of certain Eiscnstcin series and some Fourier coefficients 
attached to it. 

But one can ask more about the precise relation between these periods and central special L- values, 
known as the Ichino-Ikeda conjecture in the context of S0„ x SO„_i (cf. [IIIO]). The advantage of the 
relative trace formula approach is that it is possible to prove the explicit formula relating \B!il {ifin, "yScr)!^ 

(or I JiTr '^ '^ ((Pir, </3(7; 0)1^) and the product of certain local periods of positive type, within i-values as 
the scaling factor of these two periods. In particular, one can prove the positivity of the corresponding 
central special L- value, for example, as in [JCOl]. We will not pursue the formulation of this explicit 
relation in this paper, but instead, formulate the relative trace formulae toward it for both periods. 

1.2 Relative trace formulae and fundamental lemmas 

We briefiy describe our relative trace formula. To be simple for the introduction, we only do this for the 
case of Bessel periods, i.e., we will assume that n — m = 2r + 1 is odd. 

Let /„ G H(U„(A')) (resp. /„ e H(U,„(A'))) be a smooth function on U„(A') (resp. Um(A')) with 
compact support. We introduce a distribution 

where the sum is taken over orthonormal bases of Ajr and Aa and p denotes the action by right translation. 
As usual, we associate to /„ (E) f„i a kernel function on (U,i(/s')\U„(A') x U„i(fc')\Um(A'))^: 

C'GU„(fe') ?'eu,„(fc') 
and consider the following distribution which is formally the "sum" of J'tt^o- over all tt (E) a: 



Jifn ® U ■■ = 




{K,h[;h'2, h'^) v\h'-^h'^)dh[Ah'^. 



(H'(k')\H'(h-))^ 

The above integral is not absolutely convergent in general and need to be regularized. It turns out that 
the regular part of this distribution has the following decomposition: 

C'e[U„(A;')„g]/H'(fc') 

where [U„(A:')i-cg]/H'(A:') is the set of certain regular orbits in U„(A:') which will be discussed in Section 
4.3 and / G 7{(U„(A')) is obtained from /„ ® fm. Moreover, each summand Jc^i is an adelic weighted 
orbital integral: 

Jc'(/)= / // f{9'-'u[-\'u'yy{u[-'u'^)Au'^Au'^Ag'. 

To connect the L-function, one need to go to the GL„ x GL™ side. Let 11 = BC(7r) and E = BC(<t) 
and assume that they remain cuspidal. We define similarly a unipotent subgroup C/i'-^m+i,!'- of GL„, a 
non-reductive group H = Uir ^rn+i,i^ x GL^ viewed as a subgroup of GL„ x GL„i and a character v on 
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H (sec Section 2.1 for precise definitions). For ipn G An and ips S Aj:, consider the following general 
linear group version of the Bessel period; 

B^.r(<i5n,¥'s) := J ipn®VT.{h)v{h)-^dh. 

H(k)\H(K) 

We remark that the above integral is the usual Rankin-Selberg convolution for U„ x U,„ when r = 0, but 
not when r > 0. But in fact, it is also an integral presentation of L(s, 11 x S). 

To single out the cuspidal representations being the standard base change from the unitary groups, 
we follow [JR]. Saying that n is odd, let 



Vnim)^ J mi9l)<^9l: 'Pm{^T.)= J mi92)v{<i'^^ 92)^92 

2;,(A')GL„(fc')\GL„(A') Z;^(A')GL,„(fc')\GL™(A') 

where is the center of GL?^^' . 

Let Fn £ ■H(GL„(A)) and Fm G H(GLm(A)). We introduce another distribution 

where the sum is taken over orthonormal bases of An and A-z,. Repeat the process for the unitary groups, 
we have the kernel function ICf^^f^ and the distribution J{Fn®Fm) whose regular part has the following 
decomposition: 

Jr^^{Fr,®Fm)= ^ Jc(^) 

C6[S„(fe')rcg]/H(fc') 

where [S„(/s')rcg]/H(A;') is the set of certain regular orbits in the symmetric space S„(fc') which will be 
discussed in Section 4.3 and F G ?^(S„(A')) is obtained from F„ ® F,„. Moreover, each summand is 
an adelic weighted orbital integral: 

JdF)^ J J F{g-'u-'Cu^g)iyiu-')dudg. 

GL„(A) C/i^,„+i,i^(A') 



We prove in Proposition 4.11 that there is a natural bijection 

N : [S„(fc')rcg]/H(fc') ^ H [U„(fc')rcg]/H'(fc') 

WCV 

where the disjoint union is taken over all isomctry classes oiWdV. Hence, we pose the global matching 
condition for N(^) = C as Jc_{F) = Jc'il)- The precise conjecture of smooth matching of functions is 
proposed as Conjecture 4.12. 

As the most important and interesting problem in this kind of trace formula, wc now discuss the 
corresponding fundamental lemma. Here, again to be simple, we only state it for the unit elements in 
the following cases: n — m odd, to = 0, or n = m. 

Now let k' be a non-archimedean local field and k/k' an unramified quadratic field extension. Let o' 
(resp. o) be the ring of integers of k' (resp. k). There are only two non-isomorphic hermitian spaces of 
dimension to > over k. Let C be the pair associate to C both with trivial discriminant 
and C U~ be another one. Then will have a self-dual o-lattice Lw which extends to a self-dual 
o-lattice Ly of V^. The unitary group U+ (resp. U^) is unramified and has a model over o'. The group 
of o'-points U+ (o') (resp. U+(o')) is a hyperspecial maximal subgroup of U+(fc') (resp. U+(A:')). We also 
denote by GL„(iv') — GL„(o) a hyperspecial maximal subgroup of GL„(fc) and S„(o') :— S„(fc') n Kn- 

When ri - TO is even, we define S„,m(fc') = S„(fc') x Mi^m(fc') x ^m,iik') and \Jn,m{k') = U„(/c') x 
Mi.„i(fc). There is also a notion of regular elements in both sets and we have a natural bijection 
N : [S„,™(fc')rog]/H(fc') ^ UvFcy[Un,™(fc')rcg]/H'(fc') (cf. Scction 5.3). We propose the following 
conjecture. 
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Conjecture 1.1 (The fundamental lemma for unit elements). (1) When n — m is odd or m = 0, 

we have 



ls„(o')(3 " Cu'g)^{u )r;(detg)dwd5 



GL„(fc') C/ir,„. + i,ir(fc) 



t(C) 







l^t(o')^9'-'u[~'Cuy)v\u\~W^)du\du'^dg' N(C) = C+ e U+(fc'); 



N(C) = reU-(fc') 



where t(C) S {il} a certain transfer factor defined in (4.21). In particular, when m = 0, the second 
case of the above identity does not happen. 
(2) When n = m, we have 

ls„(o')(3~^C5)lMi,„(o')(2;.9)lM„,i(o')(5^^2/)??(dctg)d.g 

GL„(fc') 

t([C,x,2/]) I lu+(„,)(g'-icV)lM,,„(o)(^5')d5' N([C, x, y]) = [C+, z] G U+„(fc'); 

U+(fe') 







N([C,x,2/]) = [C-,z] eu-„(fc') 



where t{[(^,x,y]) G {il} is a certain transfer factor defined in (5.16). In particular, when m — 0, the 
second case of the above identity does not happen. 



We have 

Theorem 1.2. (1) The second case of the identity in both cases in Conjecture 1.1 holds; 

(2) When n — m + 1, the fundamental lemma holds if char(fc) = p > maxjri, 2} or char(fc) ~ and the 
residue characteristic is sufficiently large with respect to n; 

(3) When n < 3,m = 0, the fundamental lemma holds if char (k) ^ 2; 

(4) When n ~ m, the fundamental lemma holds z/char(fc) = p > max{n, 2}. 

Proof. (1) is proved in Proposition 4.15 and Proposition 5.12 in this paper. 

(2) is proved by Yun in [Yun09] where the transfer to characteristic is accomphshed by Gordon in the 
appendix. 

(3) is proved by Jacquet in [J92] when n = 3; (essentiahy) proved by Ye in [Ye89] when n — 2; and 
trivial when n = 1. 

(4) is proved in Theorem 5.15. 

□ 

Remark 1.3. When n ~ m + 1, the fundamental lemma is just the one proposed by Jacquet-Rallis in 
[JR]. When m = 0, the fundamental lemma is the one proposed by Flicker in [F191], which is the unitary 
group version of the Jacquet- Ye fundamental lemma (cf. [JY92]). When m > (and n — m is odd), 
the fundamental lemma is a kind of hybrid of the Jacquet-Rallis fundamental lemma and the Flicker 
fundamental lemma. We hope that there is a geometric method toward this fundamental lemma, which 
is also a kind of hybrid of those in [Yun09] by Yun and [Ng699] by Ngo. 

When 0<ri — m<nis even, we also formulate a corresponding fundamental lemma for Fourier- Jacobi 
periods. See Chapter 5 for details. 

The fundamental lemma for all elements in the spherical Hecke algebra when n = 3,m = is also 
proved, by Mao in [Mao93]. 
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1.3 Variants of Rankin-Selberg convolutions 

As wc sec in the last section, we need to consider certain periods of Bcssel and Fourier- Jacobi types on 
the general linear groups as well. We will generalize the notion of Bcssel (rcsp. Fourier- Jacobi) models 
and periods for GL„ x GL„i for a pair (r, r*) of nonnegative integers such that n = m + l + r + r* (resp. 
n ^ m + r + r*). When r = r*, they are introduced and considered in [GGP]. 

For simplicity, let us only describe the first case. Let (r, r*) be as above, we introduce a unipotent 
subgroup C/ir^m_|_i_ir* of GL„ and H = ?7ir „j_|_i it* x GL^ viewed as a subgroup of GL„ x GL,„. We 
have a character v of H which is automorphic if fc is a number field. 

Let TT (resp. a) be an irreducible cuspidal automorphic representation of GL„(A) (resp. GL,„(A)). 
We introduce the Bessel integral 



B';:^^.{s]ip^,(fa) ■■= J ipn® 'PY.{h)v{h) |det/i|^Mft, 

H(k)\H(K) 

and the Bessel period B^^^,{ipT^,ipa) ■= B'^,.,{\;ipT^,ipa) for (^^r e At^, (pa & Aa and s e C. Then B'^^, is 
the usual Rankin-Selberg convolution on GL„ x GL„i if and only if r = 0. 

For general (r, r*) and also for the Fourier- Jacobi integrals FJ^'^^-, (s; v^tt, <Pcr; we have the following 
theorem. We would like to remark that the following result is actually known by Jacquet, Piatetskii- 
Shapiro and Shalika long time ago. But since it is not recorded in the literature, we would like to write 
it down with proof just for the completeness. 

Theorem 1.4 (see Section 2.2 and 3.2 for notations). (1) The Bessel integrals are holomorphic in s and 
satisfy the following functional equation 

B';l^^,{s;(p^,ip^) = B'^, ,^{1 - s;p(w„^„)^,^). 
For tp^ G At, and (fa- G Aa such that W^'^ ~ ®vWi, and W^'^ = (E)vW~ are factorizable, 

■1 Tj «'.,r(s;W^.,W-) 



L.„{s,n,xa,) ^ 

where in the last product almost all factors are 1. In particular, there is a nontrivial Bessel period of 
TT ^ a if and only if L{^, tt x cr) ^ 0. 

(2) The Fourier- Jacobi integrals are holomorphic in s (when n > m) and satisfy the following functional 
equation 



FX^.,{s; $) = •7lC-V(1 - S;p{Wn,ni)^-„,^a;^)- 

-, ^Pa e Aa 

factorizable, 



For ip^ e A-n, •fa e Aa and $ G 5(M^'^(A)) such that W^^ = ®vWy, W^^ = ®vW~ and $ = are 



FJ^y(ip,r,fa,'^) ^ L{-,'K X a fi ) _[_[ 



'2' ^l^^ Ly{s,ny X ay® ^ivA 

where in the last product almost all factors are 1. In particular, there is a nontrivial Fourier- Jacobi period 
of TT ® a for if and only if L{^, tt x a ® t^^^) ^ 0. 

Remark 1.5. The above theorem completely confirms [GGP, Conjecture 24.1] for split unitary groups, 
i.e., general linear groups. 

It is clear that the Bessel (resp. Fourier- Jacobi) period defines an element in the space of invariant 
functionals Hom^(A) (tt (K) ct i?, C) where is a character (resp. an infinite dimensional representation) 
of H{A). It has a decomposition into spaces of local invariant functionals. The following multiplicity one 
result is a generalization of [GGP, Corollary 15.3, 16.3] for general linear groups. 
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Theorem 1.6. Let k be a non-archimedean local field of characteristic zero. Let tt (resp. cr) be an irre- 
ducible admissible representation 0/ GL,i (resp. Ghm)- T/ien dimcHom// (tt (g) cr (8) j7, C) < 1. Moreover, 
if TT and a are generic, then dinicHom^f (tt ® u ®v ,C) = 1. 

The following is an outline of the paper. 

In Chapter 2, we focus on the Bessel model and period on general linear groups. We prove the Bessel 
part of Theorem 1.4 and Theorem 1.6. The proof of the local multiplicity one result will occupy the first 
section, while we follow the idea of [GGP]. In particular, our proof includes the case r = r* which was 
left as an exercise to readers in [GGP]. In the second section, as we have said, we will give a proof for 
the global integral being Eulerian for the completeness of literature. 

In Chapter 3, we focus on the Fourier- Jacobi model and period on general linear groups. We prove 
the Fourier- Jacobi part of Theorem 1.4 and Theorem 1.6. 

After briefly recalling the Bessel model and period for unitary groups, we introduce the relative trace 
formula in Chapter 4. We prove the matching of orbits in Section 4.3 and the smooth matching of 
functions at split places. We formulate the fundamental lemma and prove the easy half in 4.4. 

Chapter 5 is repeating the previous chapter, but for the Fourier- Jacobi model and period. What 
we do more is the proof of the full fundamental lemma for U„ x U„ in positive characteristics, which is 
achieved in Section 5.5. The proof is reduced to a known combinatorial identity proved in [Yun09]. 

The last chapter is an appendix on integrals of local Whittaker functions for general linear groups. 
We collect all the results we need in Chapter 2 and 3 from existing literature. In particular, we have to 
use all kinds of auxiliary local Whittaker integrals in the theory of Rankin-Selberg convolutions. 



2 Bessel periods of GL„ x GL^ 
2.1 Bessel models 

Let /c be a local field and | |fe the normalized absolute value on fc. Let y be a fc-vector space of dimension 
n. Suppose that V has a decomposition V = X ®W ® E ® X* where W , X and X* have dimension 
m, r and r* respectively and -E = (e) with e 7^ 0, hence n = m + r -\- r* -\- \. We want to generalize 
the construction of the pair {H,v) as in [GGP, Section 13]. Let Pr,m+i.r* be the parabolic subgroup of 
GL(T^) stabilizing the flag OcXcX®W®EcV and Ur.m+i.r* its maximal unipotcnt subgroup. 
Then Ur,m+i,r-' fits into the following exact sequence: 

^ Roni{X*,X) Ur^ra+i.T* Hom(X* ,W ® E) + Hom(W ®E,X)—^Q. 

Wc may write the above sequence as: 

^ {X*Y ®X^ Ur,m+i,r' (X*)"" ® (W ® E) + (1^^ ® E"^ ) (g) X ^ ■ 

Let £x ■ X ^ k (resp. ix* ■ k — >■ X*) be any nontrivial homomorphism (if exists) and let Ux (resp. 
Ux*) be a maximal unipotent subgroup of GL(X) (resp. GL(X*)) stabilizing dx (resp. ^x*)- Moreover, 
let 

ew-{W®E) + {W'' ®E'^)^k 

be a bilinear form which is trivial on -|- and nontrivial on E and E'^ . The composition of tx + (-'x* 
and ^^r defines a homomorphism 

£ : Ur,m+i,r' {X*Y ®{W®E) + {W'^ ®E'^)®X {W ® E) ^ (1^^ ®E'^)^k 

which is fixed by {Ux x Ux*) x GL(Ty). Hence we can extend I trivially to it and define a homomorphism 
from H = Ur,m+i,r* x {{Ux X Ux') X GL(iy)) to k. Let ip : k ^ he a, nontrivial character and 
A : Ux X Ux' — !■ a generic character which can be viewed as a character of H. Let 5w be the modulus 
function of GL(Ty) with respect to the adjoint action on Ur^m+i.r* , i-e., Sw{g) ~ Idetgj]^, ~^ . Then we 



7 



can form a character v = {ip o i) ^ of H. There is a natural embedding e : H ^ GL{V) and a 

projection k : H ^ GL(W^) which together induce an injective morphism (e, k) : H ^ GL{V) x GL(W^). 
Then the pair {H, v) is uniquely determined up to conjugacy in the group GL(V^) x GL(M^) by the pair 
W C V and (r, r*). The following theorem generalizes the result in [GGP]. 

Theorem 2.1. Let k be of characteristic zero. Let tt (resp. a) be an irreducible admissible representa- 
tion oj G1j{V) (resp. G1j{W)). T/ien dime Horn// (tt (g) cr, t^) < 1. Moreover, if tt and a are generic, then 
dimcHom/f(7r a,v) — 1. 

The existence part is due to Corollary 6.3 (1). We consider the uniqueness part. The proof for k 
non-archimedcan is similar to that in [GGP, Section 15] with mild modifications for general linear groups 
and a general pair (r, r*). The proof for k archimedean should follow similarly as in [JSZIO] which we 
omit. 

Recall that we have V = X®W®E<3)X*. Let £;+ be a fc-line generated by e+. Let uq = e + e+, 
Wq = e — e+ and 

Y^X®{vo); Y*=X*®{v*); V+^V®E+. 

Let Po be the parabolic subgroup of GL(y+) stabihzing the flag : C Y C Y ® W C V"^ and 
Mo its Levi subgroup such that Mq ~ GL(y) x GL{W) x GL(r*). The group GL{V) embeds into 
GL(y+) X GL{V) diagonally. Let r (resp. t*) be an irreducible supercuspidal representation of GL{Y) 
(resp. GL(y*)) and let 

I(t,(7,t*) ■.^Indpl'-^^^T^a^T*) 

be the unnormalizcd (smoothly) induced representation of GL(V^+) of the representation t® crC>$T* viewed 
as a representation of Pq through the projection Pq — > A/q. We have the following proposition which is 
similar to [GGP, Theorem 15.1]. 

Proposition 2.2. With the notations as above and let k be non- archimedean, we have 

HomGL(y)(I [t, t*) tt, C) = HomH(7r ® u5y/ , v) 

as long as n does not belong to the Bernstein component o/GL(V^) associated to the data (GL(yo) x M, t0 
<j) and {M* x GL(Fq*),<j* (g)T*) where M (resp. M* ) is any Levi subgroup o/GL(Z) (resp. Gh{Z*)) and 
(, (resp. (,* ) is any irreducible supercuspidal representation of M (resp. AI*). Here V ~ Yq (B Z (resp. 
V ^ Z* ® Y^) with dim(yo) = dim(r) = r + 1 (resp. dim(yo*) = dim(y*) = r* + Ij. 

Proof. We need to calculate the restriction H := I(r, ct, r*) |GL(y)- By the Bruhat decomposition, there 
are six elements in the double coset GL(y)\GL(V^+)/Po whose representatives are: 

Big cell: go = l„+i. 

Medium cells: gi sends ^o to : C Yi C Yi ® Wi with Yi=Y and E+ (zYi®Wi; 
32 sends ^0 to ^2 : C F2 C F2 © W2 with Y2(lV and E+ (^Y2®W2t V. 

Small cells: 53 sends ^0 to ^3 : C ^3 C F3 ® W3 with £■+ C F3; 

34 sends ^0 to ^4 : C F4 C F4 © W4 with Y^dV and £:+ C I4 ® W4; 

35 sends ^0 to ^5 : C ^5 C Is © with ¥5 © W5 C V. 

Let (i = 0, 1,2,3,4,5) be the parabolic subgroup of GL(1/+) stabilizing = P^r^ GL(V^) and 

T^i — {t ® cr ® t*)\q-. By Mackey theory, there is a filtration C Ho C Hi C 112 = 11 such that 

Bo = clndg^^^Vo; Bi/Bo = clnd^^^^Vi ® cInd;^^(^V2; 
B2/B1 = clnd°|;(^^7r3 © cInd^^^^V4 © clnd^)}^K^ 
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where cind means the (unnormalized smooth) induction with compact support. Applying the functor 
HomQL(y)(— , tt), we have the following exact sequence: 



GL(y) 



HomGL(y) (0i=3 cIndQ 



■ HomGL(\/) (n, tt) 



HomGL(v) (ni,7r) 



is of the form Ind^Hr'^V* 



IT* (resp. where Pg* 



But for i = 3,4 (resp. 4,5), clndg. lo ui luim inup, 

(resp. Po) is some parabolic subgroup whose Levi is M* x GL(yQ*) (resp. GL(yb) x M) and ^* (resp. ^) 
is some smooth representation of M* (resp. M). By our assumption on t and r*, the second and the 
fifth (last) terms in the above exact sequence are both zero. Hence HomGL(V')(n, tt) = HomGL(y)(ni,7r). 
Applying the same functor again, we have: 



7rj,7rj ^HomGL(y) (ni,7r) HomGL(y) (Hctt) 

^ Ext^L(\/) (0L1 clndgl^^^V,, 5^ 



For i = 1 (resp. 2), clndg^^^Vi is of the form Indpt''-'' ® t* (resp. Indp^''''''r <;). Again by our 
assumption on r and t* , the second and the fifth (last) terms in the above exact sequence are both zero. 
Hence 

HomGL(y)(n,7r) = HomGL(y)(ni, tt) = HomGL(y) (Ho, tt) ^ HomGL(y)(cIndQj;'^^Vo, tt). 

Recall that Pr^m+i,r is the parabolic subgroup of GL(T^) stabilizing C X C X ®W (B E C V . Hence 
it contains Qq as a subgroup and moreover, is equal to Ur.m+i,r' x (GL(Ar) x GL(Ty) x GL(A'*)). The 
natural projection: Pq —5- GL(F) x GL(Ty) x GL(y*) induces the following commutative diagram with 
exact rows and injcctive vertical arrows: 



,GL(y)^ 



0- 



Pn 



■ GL(y) X GL{W) X GL(Y* 



■0 



iVnn 



I I 1^0 ■ 



■Qo 



■ Py X GL{W) X Py. 



0- 



r,m+l,r* 



Hom((i)o) ,A:) X Hom(A:*, (v^)) 



■0 



where Nq is the maximal unipotent subgroup of Po, Py C GL(y) is the mirabolic subgroup stabilizing 
X and Py. C GL{Y*) is the mirabolic subgroup fixing Vq. The proof of this is similar to [GGP, Lemma 
15.2]. By the diagram, we have 



(T®tT®T*)|Qo 

By a result of Gelfand-Kazhdan, we have 



clnd^-A; 



clnd^^' A* 

Uy* 



where Uy (resp. t/y.) is the unipotent radical of a Borel subgroup of GL(y) (resp. GL(F*)) satisfying 
Ux C Uy C Py (resp. Ux* C Uy C Py), and A (resp. A*) is a generic character of Uy (resp. Uy)- 

By our choice of unipotent radical, it is clear that, the pre-image of C/y x G\j{W) x Uy in Qo is the 
subgroup 



H = UrMr. X [Ux X G\.{W) X Ux-) 
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and the pull-back of the representation A ® (g> A* is just cr(5^^ ® v. Hence, by induction in stages, we 
have 

T iGL(y) ^ -r ,GL(y) r-^^~ 

clndg^^ 'ttq = cind^ ^ ""w ® ^ 
and by Frobenius reciprocity, we conclude that 

HomGL(y) (I (''', cr, T*) tt, C) = Homn ^tt ® a5y^,^ , i/^ . 

□ 

Proof of Theorem 2.1 for the uniqueness part when k is non-archimedean. We choose an irreducible su- 
percuspidal representation t (resp. r*) of GL(y) (resp. GL(F*)) satisfying the assumption in the above 
proposition. After twisting unramified characters (which still satisfy the assumption) , we may assume that 

the induced representation I ^t, (T(5^,t*^ is irreducible. Then by the above proposition and [AGRSIO], 

we have 

dimcHoniH (tt ct, z^) = dimcHomQL(y) {t, aS^, t*^ ® tt, < 1. 

□ 

Definition 2.3. A nontrivial element in the space Hom//(7r ® cr, i/) is called an {r,r*)-Bessel model of 
TT (g) cr. When r = r* , hence n — m is odd, it is just the one defined in [GGP]. 



2.2 Bessel integrals, functional equations and L-functions 

In this section, we consider the global situation. Hence k will be a number field and A is its ring of adclcs. 
Let I |a = niisMfc I I-"- ^^'^ ^ ^ ^fc' denote ky the completion of k at v. We denote o (resp. o.„) the 
ring of integers of k (resp. fc„ for v finite). For any algebraic group G over fc, we denote G„ = G{ky) the 
local Lie group for v £ Mfc. If G is reductive, we denote A{G) (resp. Ao{G)) the space of automorphic 
forms (resp. cusp forms) of G which is a representation of G(A) by right translation p. 

We define the pair {H,v) as in the local case. Hence (-x-, ^x- and £w are defined over the number 
field k] ij) is a. nontrivial character of fc\A; A is a generic character {Ux x Ux*){k)\{Ux x Ux'){^) C^. 
Let TT (resp. a) be an irreducible cuspidal automorphic representation of GL(F)(A) (resp. GL(VF)(A)). 
Then tt (resp. a) is isomorphic to a unique irreducible sub-representation A-^ (resp. Aa) of ^o(GL(T^)) 
(resp. Ao{Gh{W))). 

Definition 2.4. The following absolutely convergent integral is called an {r,r*)-Bessel period of tt ® cr 
(for a pair {H, i/)): 

B'^^^,{ip^,ipa) ■■= J ipT,{£{h))ipa{K{h))v(h)~'^dh, (p.^ e A-^, (fia & Acr 

H(k)\H{A) 

where dh is the Tamagawa measure on H{A). If there exist ip^^, ipc, such that S^^. {fTn f<y) 7^ 0, then we 
say TT ® cr has a nontrivial (r, r*)-Bessel period. 

It is obvious that B'^, .^, defines an element in 

Hom//(A) (tt ®a,v)= Honin,. (tt^ ®(Jv,^'v)- 

Since the later space has the multiplicity one property, we expect that the Bessel period is Eulerian. We 
now show that this is true. 

We can choose a basis {ui, Ur} oi X such that 
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• The homomorphism £x ■ X k is given by the coefficient of Vr under the above basis; 



• Ux is the maximal unipotent subgroup of the parabolic subgroup Px stabilizing the complete flag 
C (vi) C {vi,V2) C ••• C =X] 



• The generic character X\ux is given by 

X{u) = V {ui,2 + U2.3 + 

where 



+ Wr-l,r) 



U1.2 


Ul,3 ■ 






1 


U2.3 ■ 


■ U2.r-1 


U2,r 




1 


■ W3,r-1 


U3,r 






1 


Ur-1. 



1 



6 UxiA) 



(2.1) 



under the above basis. 
Similarly, we can also choose a basis {u*. , v^} of X* such that 

• The homomorphism ix-' '■ k X* is given by a; 1— ?■ cxv*, for some c ^ determined later; 

• Ux* is the maximal unipotent subgroup of the parabolic subgroup Px' stabilizing the complete 
flag C (v;,) C C ■ • • C {v;,,...,vl) = X*; 



• The generic character A|[/_,^, is given by 
where 

" 1 



m) 



r* .r* — 1 

1 



^r* — l,r* 

1 



-1,2 
-2.2 



-1,1 



e Ux*{A) 



(2.2) 



under the above basis. 

Moreover, we can choose a basis {wi, ...,Wm} of and {wq} of such that the bilinear form 
iw ■■(W(SE) + ®E'^) -^k is given by £w{w^) = ^vvl^j"^) = (1 < i < m) and ^iy(w^) = 1, where 
{wi , w^, } is the dual basis. Let c = ^vf(u'o)~^. 

We write elements in Gh{V) in the matrix form under the basis: 

{Wi, ...,Wm,Vl, ...,Vr,Wo,V*,,...,v'^} (2.3) 

and view GL(PF) as a subgroup of GL{V). Then the image of H{A) in GL{V){A) consists of following 
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matrices: 



h = h{n, n*, b; u, u*; i 
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72* • • • ni l 


• • • ni,,„ 

Tlr.l ' ' ' '^r.ra 


u 


nis) 
nr,o 


b 




1 











(2.4) 



where 



"-1,1 



e Honi(Ty®i;,X)(A), 



e Hom(X*,W®i;)(A), 



6 G Honi(X*,X)(A), w G C/x(A), m* G Ux'{^) and g G GL(T/F)(A). Hence m and u* are upper triangular 
matrices as in (2.1) and (2.2); the character v on H{K) is given by 

^{h) ~ ly {h (n, n* , 6; w, u*; 5)) 

= I det g\^ ^ V (^*l,2 + • ■ • + Ur-l^r + "-r,0 + "o.r* + "*',r*-l + ' ' ' + ""2.1) • 

Let = J7r.m+i,r* x (t^x X Ux*) bc a unipotcnt subgroup of GL{V), then 

^I'^.m+i !'■* i^) ~ IL {n, n*, b; u, u*) :— h {n, n* , I; u, u*; 1™)} 
and we denote du the product measure. Then we have c\h = \ det^j^"'^ dudg. Let us simply write 

Tp(u) ^ -0 (ui^2 + h Ur-l,r + ?V,0 + "-0,r- + ,r* -I ^ 1" '"2,l) 

and identify GL(V^) (resp. G'L{W)) with GL„_fe (resp. Ghm^k) under the basis (2.3). Then 

^r,r*(<^vr,<^<T) = j j <P7r (m) V<t (ff) | dct g V^(w)dwdg. (2.5) 

GL„(fc)\GL„(A) C/i. (A) 

We insert an s-variable as 



(ug) (p<T(g)| det ( 



'0(u)dud(7 



and call it the {r,r*)-Bessel integral. We are going to use the Fourier transform. Let 

■•• ni,o 

Lr+i = ( u{no, n* , b; u, u*) uq 



■■■ n 



r,0 
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be a subgroup of C/ir,,„+i,ir* . Let 

Lr = {l_ = lilr;no,n*,b;U,U*)\lr = [h.r, -Jm.r]} 

where the matrix l{lr', uq, n* , 6; u, u*) is the one obtained from u(no, n* , b; u, u*) by adding the column lr 
above the entry ni_o as in (2.4). It is clear that Lr/Lr+i is isomorphic to /c™ which may be identified 
with the set of column vector lr- By the Fourier inverse formula for {k\A)"^, we have 



(2.5) = y j j ^-aB5)¥'a(g)|det.g|l (f]e,v)i(i)dWndg (2.6) 



g M^.,„(fc\A) '='="'L^(fc)\L,. 

where n represents the clement u (n, 0,0; 1^, lr* ) with 

"1,1 ■•■ riY.rn 



"-r,l 







and we view -0 as a character on Lr{A.) through the natural quotient Lr+i{A). We also extend the 
measure on Lr+i{A) to Lr{A) by the self-dual measure on A™. Let e be an element like u{e, 0, 0; 1^, lr* ) 
where 

••• 





ei 





£,« 



then if we conjugate { by e from left to right, we can incorporate tjj (^ ^iU^r) into '0(/) and collapse the 
summation over g fc. The result is 



(2.6) 



ip^{ln3)v<y{g)\ det5|2 



'/'(Ddidndg. 



(2.7) 



g M^_i,„(fc\A) Lr{k)\L,.{ 
xMi,„(A) 



Similarly, we can introduce the subgroup Lr-i, ■■■,Li and repeat the process r — 1 more times. Then we 
finally get 



(2.7) 



(pT,{lng)'Pa{g)\ dot 5 



■0(Z)dZdnd(7. 



g M-r 



Li(k)\Li(k) 



Here, 



Li = \ i = L{li', no, n* , b; u, u*) 



^l.r—l ^l,r 



where Z(Zi; ng, n*, b] u, u*) is the one obtained from u(ni, n*, b; u, u*) by adding li on the sub-row [ui^2, wi, 
In fact, Li is the maximal unipotent subgroup of the (standard) parabolic subgroup stabilizing the flag 

C W^© (vi) cW ® {vi,V2) C ■■■ CV. 

Interchanging the order of g and n, we have 



^iTiL9n)ipa{g)\ dct g\l 



1 r + r* . 



TpiDdldgdn 



M,,„{A) a Li(fe)\Li 

|det,g|' 



ip{n)ip^) {Lg)tpa{9)i'{L)dldgdn. 



(2.9) 



M^.„(A) g 



ii(fc)\Li( 
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Now the inner double integral is just the usual Rankin- Selberg convolution (of. [JPSS83]). The following 
calculation is well-known: 



|dctg|^ ' J ipin)(p^) {l3)(Paig)±il)dldg 

GL„(fe)\GL„(A) Li(fc)\Li(A) 



g 7eC/i™(fc)\GL,„(fc) 



" 7 






■ 9 


. 1„ 




m 




. 1„ 


— m 
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In — rn 







In— ?7i 


) 



^„{g)\Actg\'-^dg, (2.10) 



Uir^ (fc)\GL„(A) 



where ?7im is the standard maximal unipotcnt subgroup of GLm and Wf is -(/'-Whittakcr function. 
Factorize the integral over ?7im(A), we have 



(2,10) . / / 



£/i™(A)\GL„(A) Ui^(k)\Uiii 



W 



u 

1 n — m 



^7lTT^{A)\GL^(A) 



/ 

)\G1 

/ 

)\G1 



[/i™(A)\GL^(A) 



5 

In — m 

5 

l^i- ?n 

5 

In— ?n 



Va{g)du\dctg\l " d.g 

V'(w)<^CT(ff)du |detg|^ ~dg 



C/i™(fc)\C/i™(A) 

<^(.9)|det.gir'^d5 



g 

In— m 



n W^'^(g)|det5|; dg. 



(2.11) 



C/i™(A)\GL„(A) 

Plug (2.11) into (2.9), we have 
(2.9) = 



C/i™ (A)\GL„(A) M^_„(A) 



g 
X 1^. 

'Lji—jji—j- 



WS(g)\dctgf^ dxdg. (2.12) 



We denote the above integral by ^'r(s; W^^^ which is absolutely convergent when 3?(s) ^ 0. If we 

assume that W^^ = ®vWy and W^^ = ®vW~ are factorizable and let 



9v 
1„_™ 



W^."(5«)|det ^ dxvdg^. 



4',,,(s;W^„,W-) = j J 

(7ir,._„\GL™.„ M^,,„,„ 

Then in summary, we have for 3fi(s) large, 

where £ >V(7r.u, -01,) (see Section 6 for notations); W.^^ G yV((T„,V'«) and for almost all finite places v, 
Wv, W~ are unramified satisfying VK„(1„) = W~{lm) = 1- 

Now we discuss the functional equation of the Bessel integrals. First, let us introduce some Weyl 
elements: 



Wi = li; Wn 



1 
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Wc define t the outer automorphism of GL„ and GL,„ by = g'' := *g ^. Let ^p-nig) = tpTr{g'') = 
(pTriwng') and (^(g) = ipaig') = 'Paiwyng'')- Then 

= j j ^Am')^M')\dcig'\'i^^^~±{u)duAg 

GL„(fc)\GL,„(A) C/j. (fc)\C/ir,„ + i.i.- (A) 

= J J 'f^{u'g)'?^{g)\detg\~''^"^~^^~i}_{u)dudg 

GL„(fe)\GL„(A) C/i. „ + i WXt^i'-.^ + i,!-* (A) 

= J j {p{wn,,n)'^){'Wn,mu''Wn^„ig)(p^{g)\ det g]^'"'^'^'^ ^ V^(u)dMd5. (2.13) 



Since we have Wn,mUiT 

C^i'-*,m+i,i'- ^i^d V^('"^n,inw''w„,m), we have 

(2.13)= / / (p(ifn,m)<^) (m)<A^(.9)I detgl^ " = '0(li)dudg 



= Kr.,r(l - s; p(w„,m)^, ^). 
In summary, we have the following 

Theorem 2.5. The Bessel integrals are holomorphic in s and satisfy the following functional equation 

B'^^^,{s]Lp^,ip„) = B'^.. ,^{1 - s;p(ui„,„i)^,^). 

Let Wf(g) = Wfiuong') € W{n,i^) and similar for the one on GL^. IfW^^ = (giyWy and W^^ = ^yW- 

are factorizable, then W^^ = ®vWy and W^^ = ®vWv are also factorizable with Wy{g) ~ Wv{wng'') 
and similar for . Then for ^(s) large, 

S,V(s;¥.^,¥.,) = 4',(s;M^^t,M^^J= [] ^yAs;Wy,W-); 

In particular, when r ~ r* (hence n — m is odd), the (r, r)-Bessel integral itself has a functional equation. 

By Proposition (6.1) and (6.2), wc have the following theorem, which confirms [GGP, Conjecture 
24.1] for the Bessel periods of split unitary groups, i.e., general linear groups. 

Theorem 2.6. (1) Let t: (resp. a) be an irreducible cuspidal automorphic representation of Gh{V){ A) 
(resp. Gh{W){h) ). For any (r, r*) such that r + r* = n — m — 1 and the automorphic representation v 
introduced above, we have, for (/s^ € A-,,: and Lpa G Aa such that W^^ = ^yWy and W^^ = t^dyWy' are 
factorizable, 

1 , -TT ^^yAsiWy^W-) 



^r,r*(¥'7r,<<5^) = i(;j,7r X cr) J| 



.eM, Lyis^TTyXay) 

where in the last product almost all factors are I, and the L -functions arc the ones defined by Rankin- 
Selberg convolutions (cf. [JPSS83]). 

(2) There is a nontrivial Bessel period of n ® a if and only if L{^,tt x <t) 0. 
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3 Fourier-Jacobi periods of GL„ x GL^ 
3.1 Fourier-Jacobi models 

Let fc be a local field and V a /c-vector space of dimension n > 0. Suppose that V has a decomposition 
V = X (B W (B X* where W, X and X* have dimension m, r and r* respectively. Then n = ni + r + r* . 
Let Pr,m.r' be the parabolic subgroup of G'L[V) stabilizing the flag C X C X ®W <ZV and Ur,m,r' its 
maximal unipotent subgroup. Then Ur,m,r' fits into the following exact sequence: 

O^Hom(X*,X) Ur,ra,v ^ Hom(X* , VK) + Hom(W", X) ^ . 
We may write the above sequence as: 

^ {X*Y ®X^ C/r,m,r- ^ {X*Y ® W + ® X Q . 

Let Ix '■ X k (rcsp. ix* ■ k — > X*) be any nontrivial homomorphism (if exists) and let Ux (resp. 
Ux*) be a maximal unipotent subgroup of GL{X) (resp. GL{X*)) stabilizing £x (rcsp. ix*)- Then the 
above exact sequence fits into the following commutative diagram: 

>- {X*y ® X ^ C/r,m,r- ^ (gl X + {X*)'' ® W ^ 

^ k ^ H(W^^ + W) ^ W"" + W ^ 

which is equivariant under the action of Ux x Ux* x GL{W), where H(iy^ + W) = k + W'^ + is the 
Heisenberg group of W'^ + W whose multiplication is given by 

/, V N/, V N , . , ^iM ~ w^jwi) V , V 
{ti,Wi ,Wi)(t2,W2 ,W2) = Ul+t2H ^ +'W2,Wi+W2 

Given a nontrivial character i/' : fc — >■ C^, there is a unique infinite dimensional irreducible smooth 
representation oj^, of H(iy^ + W) with central character ip. We choose the following model. Let S{W'^) 
be the space of Bruhat-Schwartz functions on . For <i> G 5(VK^), let 

for all {t,w'^ ,w) € li{W^ + W). Moreover, if we choose a character /i : fc^ ^ C^, we have a Weil 
representation u^, of GL(W^) on S{W'^) by 

{oj^{g)<i>) {w') ^ fi{dctg)\dct g\l^w' -g) 

where g G GL(iy) acts on W'^ by (w^ ■ g) w = (5'w) for all w G W. They together form a representation 
uj.^,,f_,, of Ur,m,r' X GL(M^) through the projection Ur^m^r' — > H(iy^ + W) and hence a representation of 
H := C/r,m,r* X {Ux X Ux* X GL{W)) by extending trivially to Ux x Ux*- As in the Bessel model, we 
choose a generic character A : Ux x Ux* C^. Then we define the representation 

of H which has the Gelfand-Kirillov dimension to. Wc also define 

As in the Bessel model, we have an injective morphism (e, k) : H ^ GL{V) x GL(VF). Then the pair 
{H, Vfj) is uniquely determined up to conjugacy in the group GL(F) x GL(M^) by the pair W CV, (r, r*) 
and fi. The following theorem generalizes the result in [GGP]. 
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Theorem 3.1. Let k be of characteristic zero and non-archimedean. Let tt (resp. a) be an irreducible 
admissible representation of Gh{V) (resp. GLi{W)). T/ien dime Hoin//(7r (8) cr (g) i^, C) < 1. Moreover, if 
TT and a are generic, then dime Horn ff(7r (g) cr (g) i^, C) = 1. 

The existence part is due to Corollary 6.3 (2). We consider the uniqueness part whose proof is similar 
to that in [GGP, Section 16] with mild modifications for general linear groups and a general pair (r, r*). 

Recall that we have V ^ X ®W ® X* . Let r (resp. r*) be a supcrcuspidal representation of GL(X) 
(resp. GL(X*)) and let 

I (r, a,T*):= ^^A^p}^], (r ® a ® t*) 

be the induced representation. We have the following proposition which is similar to [GGP, Theorem 
16.1]. 

Proposition 3.2. With ip a fixed additive character of k which is non-archimedean, we have 

HomGL(y) (l (t, ct/^"M^, t*^ ® n, wy,^^ = HomH(7r ® cr, i/^) 

as long as tt does not belong to the Bernstein component of GIj(V) associated to the data (GL(X) x M* , r <^*) 
and (A/ x GL{X*),<; (g) r*) where M (resp. M* ) is any Levi subgroup ofGh{X + W) (resp. GL(iy + X*) ) 
and (resp. ) is any irreducible supcrcuspidal representation of M (resp. M*). Here, ujy,^ is the Weil 
representation of G1j{V). 

Proof. Let Mp(M^ + W"^) be the C^-metaplectic cover of the symplectic group of the symplectic space 
W + W^. We choose the Lagrangian and an additive character -0, and hence get the a model of Weil 
representation ojw,^!- We fix a homomorphism GL(VF) ^ Mp(iy + W^) (by choosing fi to be trivial) 
hfting the embedding GL(M^) ^ Sp(VF + W^) such that 

{uwAgW {w') = \dctg\l^w'g). 

We consider another symplectic space Vi + V^^, where Vi = X + {X*^ + W and = W'^ +X* + X^. 
We choose the Lagrangian Vi C W"^ and consider the mixed model of the Weil representation wvi.^ of 

M-p{Vi+V^). This model has a realization on the space S (^{X + (X*)^)"") (SSiW) = S{W''+X*+X''). 

We choose a lifting of the embedding GL(T^) ^ Sp(Vi + V(^) such that the following diagram commutes: 

Mp(W + W^)^ ^ Mp{Vi + V^i^) 




The parabolic subgroup Pr.m,r- of GL(y) is contained in the parabolic subgroup P{Vi,W) of Sp(Vi +V/) 
stabihzing the flag C X + {X*Y C X + {X*Y + + . Elements in Pr,m,r* can be written 
as p = p{n,n*,b;h,h*;g) with n e Rom{W,X), n* G llom{X*,W), b G Hom(X*,X), h £ GL{X), 
h* e GL{X*) and g G GL(M^). We have the following formula for the mixed model: 

(a;yi,^(p(0, 0, 0; h, h*; {w\x^,x^) = \ det hgh*~^\l<i>{uj^ g, h*-^x^ ,x^h); 

[uv.Mn, n\b- Ir, Ir' ; 1™))$) {w\x\x^) = V (x^ {b + w\n*{x^))) $ (u)' + n'{x^),xKx^) 

for {w^ ,x\x^) G +X* + X^ . We denote by uv^^p = ^Vi,^I;\gi.(v)- Then there is a P^^^.r* -equivariant 
map: 

ev : LOv,,p — > |dctx|^ '^uJw+x-,^^t^x+w,i, ® |detx'|fc ^ 
'^{w\x\x^) ^ ($(w^x^O),$(w^O,a;^)) . 
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The kernel of ev is generated by the Schwartz functions $i (g) $2 (8 $3 S S{W^) S{X*) S{X^) such 
that $2 (resp. $3) is compactly supported on X* — {0} (resp. — {0}). Since lJt^ = uj^:^, we have 
the following exact sequence of smooth Pr,m,r* -representations: 

^cInd^_^^__^^(P^^QL(iv)xP:„.)l'^°*-^lfe ® Idetx-lfc ' 



Idetxife ® c^w+x^' ® (^'x+w^i, ® |detx-|fe ' 

where Px (resp. Px*) is the mirabohc subgroup of GL{X) (resp. GL{X*)) stabilizing £x (resp. £)^,) 

1 

and oJw,^! is a representation of Nr,m,r* x GL(M^) which coincides with Vf^^~^S^. 
' '1 
Tensoring the above sequence with T(E)<Jfi~^S^^T* and inducing to GL{V), we get an exact sequence 

of GL(F)-rcprcscntations: 



where 



^ I (r, (7^-1,5^, T*) ^B®B* ^0 



A = cInd^J;_^2^J^^(p^^(.L(;^)^P^.^) (^Ipx) Idetxife' <S)ujw,4, ® (r*|p^. ) |detx* |fc ' ; 
B = Indp^^^^'^^r|detx|^ (oJw+x*.i> ® crf^^^^w ® '^*) > 
S* = Indp_^^^p^^ (u^^^^(g>T(g)a|I''^^|^,'^ (g> T*\detx*\^^ ■ 
By our assumption on tt, wc have 

HomGL(y)(B,i) = HomGL(y)(S*,^) = ExtJjL(y) (S, i') = ExtGL(y)(S*, i=) = 0. 

Hence 

HomGL(V) (l {T,<TH~^d^,T*^ ,TT^ 

'^^^'^N^,„,^,.xiP^xGh{W)xP^,} (^lPx)ldctxlfe' (^*|Px-)|dctx-|fc %7rj 

Again since t and r* are irreducible supercuspidal, we have 

,GL{X), 



Px 

and by induction in stages, we have 



rldctxll „ =cInd;;^^-^^A; r*|detx- 



cInd;;^(^^*)A* 



^^^'^N,,^i,xiPxxGL{W)xP^,)i^\Px)\'^(i^x\i Kiwiy,^ ® (t*|p^.) |detx*|fc ' = clnd^ V®z/^. 
Thus the proposition follows by the Frobenius reciprocity. 



□ 



Proof of Theorem 3. 1 for the uniqueness part. We choose an irreducible supercuspidal representation r 
(resp. r*) of GL(X) (resp. GL(X*)) satisfying the assumption in the above proposition. After twisting 

unramified characters, we may assume that the induced representation I (^,a^,~^5^,T*^ is irreducible. 

Then by the above proposition and [GGP, Theorem 14.1(iv)], we have 

dimcHom// {tz ® a ®'u^,C) ^ dimcHomGL(\/) {j, ct^^'^'^m/i '''*) ® ti", < 1- 

□ 
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Definition 3.3. A nontrivial element in the space Honi(7r (g) a ^ i/fj_,C) is called an (r, r*) -Fourier-, J acohi 
model of TT (g) cr. When r = r*, hence n — m is even, it is just the one defined in [GGP]. 



3.2 Fourier-Jacobi integrals, functional equations and L-functions 

Let fc be a number field, {H^v^) be a pair associated with nontrivial ip : k\A C^, /i : k^\A^ — > C^, A 
a generic character {Ux x Ux'){k)\{Ux x C^x*)(A) — !• and realizing on the space SiyV^ {K)). 

For any $ G <S(VK^(A)), we define the theta series 

which is an automorphic form oi H. Let tt (resp. a) be an irreducible cuspidal automorphic representation 
of GL(F)(A) (resp. GL{W){A)). 

Definition 3.4. If n > m, the following absolutely convergent integral is called an (r, r*) -Fourier- J acohi 
period oi n ® a (for a pair (i7, v^)): 



ip^{e{h))(pa-{K{h))0-:T; J -,{h,^)\ dot h\j^^ dh, ip^ e A^, fa ^ Aa 



H{k)\H{A) 



If there exist ip.^, (p„ and $ £ S{W'^{A)) such that JiT^'^,{ipTr,'Pa]^) 7^ 0, then we say tt ® cr has a 
nontrivial (r, r*)-Fourier-Jacobi period. The case m = n will be discussed in Remark 3.6. 

It is obvious that JiJ^'^t defines an element in 

Hom^(A) (tt (g) tj (g) zy^, C) ^ Hom/j-^ (tt^, CTi, (g) C). 

Since the later space has the multiplicity one property, we expect that the Fourier-Jacobi period is also 
Eulerian. We now show that this is true. 

We choose basis {vi,...,Vr} (resp. {vl, ...,v*,}) for X (resp. X*) in the same way as in Section 2.2 
(with c — 1). Wc also choose a basis {wi, Wm} of W with dual basis {wi, w^} of W'^ . We identity 
GL{V) with GL„jj and hence GL(iy) with GL„i^fc via the basis: 



{Wi, ...,Wm,Vi, ...,Vr,V*,,...,vl} . 

Then the image of H{A) in GL„(A) consists of following matrices: 



(3.1) 









n* ■ • • ni l 




9 








7^1,1 • • • ni^rn 






h{n, n* , b; u, u*;g) = 


Ti^\ • • ■ ri^jji 


U 


b 






u* 



(3.2) 



where n, n* , b, u, u* and g are similar to those in Section 2.2, but without entries related to wo- Let 
Uir- m i"-* = Ur,m,r* X {U X X Ux') be & unipotcnt subgroup of GL„, then 



UiT „ ]^r* (A) — {u — u (n, n* , 6; u, u*) := h (n, n*, I; u, u*; Im)} ■ 
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For $ e S{W"^{A)), we have 

(fc'^(u)3>) (W^) =-0 ("1,2 H h Wr-l,r + K^r' + U*.^r'-1 "I 1" "2,1 + ""^ ' ["l,r* i '^m,r*]) 

^{W^ + K,!, ...,nr,m])- 

For simplicity, we denote ip{u) = ?/'(mi,2 + • ■ • + Ur-i,r + ^r,?-* + u*, + • • ■ + U2 i). Then 



J'J,':;, ((^., $) - TJ,":;, (-; <&) 



where 



I 



<P7r(M)¥'a(5)%,A,^-i(M,*)|detg|^ ' dudg. 

There are two cases. 

Case 1: r > 0. In this case, we have 



(3.3) 



where = u{nr, 0, 0; 1^, Ir*) and 



••• 



We define the subgroup ir+i of „ j^r* in the similar way as before just by taking the matrix n = 0, 
then we separate n from u and summate the last row in n over fc™. We get 



(3.3) 



/ / / 



g M,_i,„(fc\A)L,+i(fc)\L,+i( 



(^^(n^/ng)(^<,(5)A(;) (c^;^_^_i(l5)$j (nj|detg|2 ' dn,,dWnd5 



(^^(i7i5)(^^(5)$(n^.g)V^(/)/,t(detg) Vetffll"^ ' d/drid.g (3.4) 
where in the last line, is the last row of n. If we repeat the process (2.6), (2.7), (2.8) and (2.9), then 



g M,_i,™(fe\A) L,+i(fc)\L,+ i( 
xMi,„(A) 



(3.4)= j y M(detg)"Vct5ll ' 

M,.,„,(A) g Li(fc)\Li(A) 

By the classical argument for the Rankin-Selberg convolution, 

£/im(A)\GL,„(A) Mr-i.„(A) Mi,™(A) 



ipin)Vn) mMgMRrWMdgdn. (3.5) 



^1 



g 

X Ir-l 

2/010 
1,.. 



Wl{gMy)fi{detgr'\detg\l ""^"^ dydxdg. (3.6) 
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If we denote (3.6) by ^r{s', , W^^ ® ^; $), then it is absolutely convergent for 3R(.s) > 0. Moreover, 
if W:^^ =®vW^, W^^ = (E,yW- and $ = are factorizable, we have 



where 



g„ 

Xu lr-1 

1 

1^. 



(7im „\GL„,„ M,._i,„,„ Ml. 



igv)^viyv)^J'vi<ict gy) ^jdetg^l^ ^ dyydxydgy. 



Case 2: r = but r* > 0. Let P,„ be the standard mirabolic subgroup of GL,„ consisting of 
(invcrtiblc) matrices whose last row is ~ [0, ...,0, 1] G fc™. Then 

=A(u)M(dctg)|det.g||$(0) + /i(det 5)1 detail ^ A(u) (e™7g) 

7eP™(fc)\GL„(fc) 

where the first term (for /^-i ) contributes to JiJon-mi^' 'PTT,'Pa', ^) since lyS^ is a cusp form. Hence, 

</'7r(gw)¥'fT(.9)$(em5)V'(emffn*e*„)V^(u)/i(detg)" jdetgl^ ^ dgdu (3.7) 

where w = w(— , n*, — ; — , u*) since r = and e*„ = '[1, 0, 0]. Applying the Fourier inverse formula to 
if a, we have 



(3.7) = y j (p,(52i)W^^„(5)$(e™ff)V^(e™3n*e*J^(u)M(dctg)-Vct5ir ^ dgdu. (3.8) 

M C/im(fc)\GL„(A) 

Factoring the inner integral through Uim[k)\Uim{A) and incorporating this unipotent part into u, what 
we get is the integral over Ui^ {k)\Uin (A) where Uin is the standard maximal unipotent subgroup of 
GL,i. Moreover, if we change the order of g and u G C/in (fc)\[/in (A), all terms involving ijj will form a 
generic character ip of C/in (fc)\C/in (A): 

%jj{u) = ip {uia H h u„_i,„) . 

In all, 

(3.8)= j j ^,(ug)^iy^j5)*(e™g)/^(det5)^i|det5ir'^dudg 

;7i™(A)\GL„(A) C/in(fe)\C/i"(A) 

/ <:( I ) VKX(3)$(e™g)A.(det3)-i|dct.9ir'^d5. (3.9) 

C/im(A)\GL„(A) 
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If we denote (3.9) by ^'o(s; , W^^ ®n ^; $), then it is absolutely convergent for 3R(.s) > 0. Moreover, 
if W^^ = ®vWv, W^^ = ®vW~ and $ = ®v'^v are factorizable, we have 



where 



g„ 

1 n — rn 



(ff«)*t;(em5.„)M„(det5^) ^|detg„|^ ^ dg„. 



C/im_„\GL„,„ 



Now wc discuss the functional equations of the [r, r*)- Fourier- J acobi integrals Jil^'^,{s; ifTr , ipa', ^) 
First, there is a linear map — : iS(M^^(A)) — > S{W{A)) given by 



q>{w^) = y $ 



If we identify W with W"^ through the basis {wi, ...,Wm}, then — is an endomorphism of iS(M^^(A)). 
Consider the group isomorphism tr*,r '. H ^ H given by Lr',ril!dj) = Wn,mlll Wn.mg'' ■ Then for any 
h £ H{A), we have the following commutative diagram which can be checked directly: 



(3.10) 



^■^.^-l{'^r*.r(h)) 



5(W^^(A)) ^5(VF^(A)) 



Then 

Tj';'.!;,{s;ip^,ip^]^) 



<y5^(M')'^a(5')%,A,M-i(M\«')|det5|A'+^+ = dudg 



dudg 



a a 



(p(w„,m)(p^) (u5)cp<,(g)6':^ j_^_i (ir.,r(u5),$) Idetgl^ = = dwdp (3.11) 
But by the Poisson summation formula and (3.10), 

= ^ AU (ir-,r(M)) (W«) 

= X! Au (^a;^,^(w5)$^ (w') 

u;lt6H'v(fc) 



22 



Hence, 

(3-11) J {p{wn,,n)(p^) {ug)'P^{9W4',\fj.ili3,^)\'dc^9\A ' ^ dwd^ 

In summary, we have the following 

Theorem 3.5. Let n > m, the Fourier- J acobi integrals are holomorphic in s and satisfy the following 
functional equation 

Let Wt{g) = Wt{wng') 6 W{l:,ii) and similar for the one on GL^. // W^^^ = ®^W^, W^^ = ®^W- 

and $ = ®^<i>t, are factorizable, then W!p„ = O^IV^, = ®i,Wv and $ = ®v'^v o,re also factorizable 

with Wv(g) VK„(u)„g'), W7(5) = W^iwrng') and = $„. Then for 5R(.s) large, 

veMk 

In particular, when r = r* (hence n — m is even), the (r, r)- Fourier- J acobi integral itself has a functional 
equation. 

Remark 3.6 (The case n = m). In this case, V = W and H = GL„. Wc will see that this is exactly 
the case of Rankin-Sclbcrg convolution for GL„ x GL„. For simplicity, we assume that ttM a ® is 
unitary. We fix a basis {vi, v„} for V and identify 1/^ as the set of row vectors of length n. In this 
case, there is no choice of A anymore. 

For any $ G <S(F^(A)) and any character s-C^ such that /i • x is unitary, we define 

which is absolutely convergent when K(s) > 1. It has a meromorphic continuation to the entire complex 
plane which is holomorphic at s = ^ (cf. [JS81]). For any holomorphic point s, it is in ^(GL„_fc) with 
central character x~^- Moreover, 

o;aI; g, x) ■■ (^M, ^(^^(A))) A (GL„^,) 

is GL„(A)-equivariant. We denote Z„ the center of GL„, x-k (resp. x<t) the central character of tt (resp. 
a) and let 

jP7"o"S(s;<^^,<^^;*) = j v^{g)vAg)s^^^-As;g,^,x.-Xa)Ag (3.12) 

Z„ (A) GL„(fc) \GL„ (A) 

be the usual Rankin-Selberg integral (cf. [JS81, JPSS83]) which is absolutely convergent at any holo- 
morphic point s. The Fourier- J acobi period JiT^Q{(pTr,ipcr;^) := JiJ'QQ{'^;ipTr,'Pcr]^) defines an element 
in 

HomGL„(A) (tt ® (T (g) j^, C) = IIom_H-„ (tt^ (g) (g) 17^, C). 

veMk 
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Unfolding 0^ i , we see that 



(3.12)= j <p^{g)ip„{g)<^{eng)^x{detg) ^\detg\ldg 

P„(fc)\GL„(A) 



/ <j5X„(5)'i>(e„5)/i(detg)-Vctff|id5. (3.13) 

f7i.i(A)\GL„(A) 

If we denote (3.13) by '^q{s; W^^ , W^^ (g)^"^; $), then it is absolutely convergent for ^{s) > 0. Moreover, 
if W"^^ = ®vWv, W^^ = ®vW~ and $ = (g)t,$t, are factorizable, we have 

where 

*i,,o(s; ®m;;^$,„) = y" W:„(5„)iy„"(g„)$„(e„g„)//„(detg,„)"^|det5r„|^dg„. 

(7i>.,„\GL„,„ 

Moreover, we have the following well-known functional equation 

JUq^o (s; , ; $) = JlT'o'^S (1 - s; ^, ^ ; $) 

and Jl7'q'^q(s; (ySTT, <y5(T; 'f) will have possible simple poles at s = — icr and s = 1 — icr with a real only if 
TT = (g) ^1 det 

By Proposition (6.1) and (6.2), wc have the following theorem, which confirms [GGP, Conjecture 
24.1] for the Fourier- Jacobi periods of split unitary groups, i.e., general linear groups. 

Theorem 3.7. (1) Lett: (resp. a) he an irreducible cuspidal automorphic representation of Gh{V){A) 
(resp. G1j{W){A)) ^. For any (r, r*) such that r + r* = n — m and the representation introduced 
above, we have, for (^^ € A-,^, ipa E Aa and <l> £ 5(VK^(A)) such that W^^ = i^ivWy, W^^ = <SivW.^ and 
$ = (g)$„ are factorizable, 

Ji7^;. ((^^ , ; $) = i(:r , TT X cr (g ^ ) 1 1 



where in the last product almost all factors are 1, and the L- functions are the ones defined by Rankin- 
Selberg convolutions (cf. [JPSS83]. 

(2) There is a nontrivial Fourier- Jacobi period of ® a for if and only if L{^,tt x a ® t^^^) 7^ 0. 



4 A relative trace formula for U„ x U^: Bessel periods 
4.1 Bessel models and periods 

Let k' be a field and k/k' be a separable quadratic extension which may be split. Let r be the unique 
nontrivial involution of k fixing k' . We denote by Tr and Nm the trace and norm of k/k' , respectively. Let 
k^ ~ {x E k \ x'^ = —x}. We fix a nonzero element j E k^ once for all and define Tr(a;) = j{x — x'^) e k'. 

If k' is a local field or a number field, we denote 77 the character associated with k/k' via the class 
field theory. Let Her„(fc/fc') be the set of all 71 x n matrices satisfying ^g'^ = g and Her„(fc/fc')^ = 
IIer„(fc/fc') n GL„(/c); IIer~(fc/fc') be the set of all nxn matrices satisfying 'g'^ = —g and Her~(fc/fc')^ = 

^When n = m, to prevent the occurrence of a pole at s = ^, wc assume that the character Xtt 'S' Xo- ® A*^^ is unitary for 
simplicity. 
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Hcr^(fc/fc') n GL„(fc). For a vector space X over k, we denote the vector space over k with the same 
underlying set of X but the action of k twisted by r. We also simply denote X^ by X. All hermitian or 
skew-hermitian spaces over k are defined with respect to r and are assumed to be non-degenerate. 

Let us briefly recall the definition of Bessel models and periods for unitary groups in [GGP]. First, let 
us consider the local situation, hence k' is a local field. Let V he a hermitian space over k of dimension n 
with the hermitian form (— , — ) and W C V a subspace of dimension m such that the restricted hermitian 
form (— , —)\w is non-degenerate. We assume that the orthogonal complement W-^ = E ® X (B X* such 
that E is a, non-degenerate line and X, X* are isotropic, perpendicular to E and of dimension r. Hence 
n = m + 2r + 1. The hermitian form restricted on W (resp. X © X*) identifies W (resp. X*) with W 
(resp. X). We denote V{V) (resp. U(M^)) the unitary group of V (resp. W) which is a reductive group 
over k' . Let ^j^-^ be the parabolic subgroup of U(y) stabilizing X and [/^ its maximal unipotent 
subgroup. Then fits into the following exact sequence: 

^ Ar ^ Krn+l ^ Homfe(W^ ®E,X)^0 

where /\^ X C Xr ^ X = Homfe(X, X) consists of homomorphisms b such that 6!)^ = —h. Here 6^ is just 
: — > Xr but viewed as an element in llomk{Xr,X). 

Let i'j^ : X ^ k he any nontrivial homomorphism (if exists) and let U'x he a maximal unipotent 
subgroup of GL{X) stabilizing f-^. Let i^^ ■ k W ® E he a nontrivial homomorphism such that its 
image is contained in E. Hence we have a homomorphism 

£' : f/;„,+i — ^ Homfe(VF © E,X) (^"'Ij^^^ 

which is fixed by U'^ x U(VF). Hence we can extend £' trivially to it and define a homomorphism from 
H' = X {U'x X \J{W)) to k. Let ip' : k' ~> be a nontrivial character and X' : U'^ ^ be a 

generic character. We define i/' = {t// o Tr o £') A' which is a character of H' . We have an embedding 
(e, k) : H' ^ U(y) xU(T4^). Then up to V(y) x U(M^)-conjugacy, the pair {H' , v') is uniquely determined 
by C T^. 

Let TT (resp. a) be an irreducible admissible representation of U(V^) (resp. \]{W)). A nontrivial 
element in Hom//'(7r ® a, v') is called a Bessel model of tt (g) cr. In particular, when k/k' is split, then the 
Bessel model is just the (r, r)-Bessel model for general linear groups introduced in Section 2.1. We have 
the following multiplicity one result. 

Theorem 4.1. Let k he of characteristic zero and tt, a as above. Then dimcHom/// (tt (x) a, v') < 1. 

Proof. If k is non-archimedean, this is due to Aizenbud-Gourevitch-Rallis-SchifFmann [AGRSIO] for 
m = n — 1 and to [GGP, Section 15] for general n, m. If k is archimedean, this is due to Sun-Zhu [SZ09] 
and Aizenbud-Gourevitch [AG09] for m = n — 1 and to Jiang-Sun-Zhu [JSZIO] for general n,™. 

□ 

Now, we discuss the global case. Let k/k' be a quadratic extension of number fields. We have the 
notions, o', k'^,, o'^, for v' £ M^', and A', -0' similar to those for k. Let A' : C/j^ (fc)\[/^(A) — >• be a 
generic character. Then we have the pair (iJ', v') in the global situation. 

Let TT (resp. a) be an irreducible tempered representation of U(F)(A') (resp. \]{W){A!)) which occurs 
with multiplicity one in the space Ao(\J{V)) (resp. Ao{\][W))). We denote by A-„ (resp. Aa) the unique 
irreducible tt (resp. cr)-isotypic subspace in Aq{\]{V)) (resp. Ao{'^{W))). 

Definition 4.2. The following absolutely convergent integral is called a Bessel period of tt © cr (for a 
pair {H',v')): 

B';'iip^,ipa) J ip^{e{h'))ip^{K{h')y {h')-^dh' , G A^.ifa e A^. 

H'(k')\H'{A') 

If there exist ip-^, ip„ such that B^^ {'^■m ^a) ^ 0, then we say i: ® a has a nontrivial Bessel period. 
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It is obvious that Bf, defines an element in 



RoniH' (A) (tt «) cr, ) = RouiH'^^ (tt^' <E) a^,' , v[,, ) . 

'u'eM,,, 



In order to decompose the distribution J"^ in the next section. We choose a basis {wi, of X 

such that 

• The homomorphism £^ : X — > fc is given by the coefhcient of Vr under the above basis; 

• U'x is the maximal unipotent subgroup of the parabolic subgroup P'x stabilizing the complete flag 
OC C {vi,V2) C ••• C {vi,...,Vr) =X- 

• The generic character A' is given by 

\'{u') = V' (t? «.2 + ^4,3 + ■ • • + 

under the above basis. 

We denote by {wi, Wr} the dual basis of X. We also choose a basis {w\, w™} of and {wo} of E such 
that the homomorphism I'y^ : fc — ^ © i? is given by a i-^ awQ. Let /3 ~ [(""^i: =i G Herm(fc/fc')^j 

/3o = (wo, ifo) G fc'^ and 



/3o 



GHer™+i(fc/fc')^ 



(4.1) 



We identify U(l^) (resp. U(W^)) with a unitary group of n (resp. m) variables U„ (resp. Um) under the 
basis {wi, Wr, wi, Wm,wo, Vr, vi} and view as a subgroup of U„. Let U[r ^^^i = U^, xi f/^ 
be a unipotent subgroup of U„. Then the image of iJ'(A') in U„(A') consists of the matrices h' = 
h'{n\b']u';g') ^ u'in' ,b';u') ■ g' where g' e U,„(A'). Here, 



u' =2^{n',b';u') 



Ir n' Wr ( b' + 
lm+1 



l-m+1 



e f/{..™+i(A') 



where n' G Mr,m+i(A), 6' G IIer7(A/A') (similarly defined as for fc/fc'), u' G ?7x(A); n^, Si'^^if^ 
and u' = Wr ^u''^'~^Wr. The character t^' on H'{A') is given by 

(4.2) 



ly'ih') = i.'(/i'(n',6';7.';g')) = '^'i^') = ^(m') := V-' (Tr (u'l,^ + • • ■ + + 
Then the Besscl period 

/ 



u,„(fc')\u™(A') c/;. „+i(fc')\c/;r,™+i(A') 



4.2 Decomposition of distributions 

We describe the relative trace formula on the unitary pair relating the Bessel periods. Recall that tt (resp. 
a) is an irreducible tempered representation of U„(A') (resp. Um(A')) which occurs with multiplicity one 
in the space „4o(U„) (resp. Ao(Um))- For simplicity, we further assume that the standard base change 11 
(resp. E) of TT (resp. a) is an irreducible cuspidal automorphic representation of GL„(A) (resp. GL„,(A)). 
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Let /„ g ■H(U„(A')) (rcsp. e 'H(Um(A'))) be a smooth function on U„(A') (resp. U„i(A')) with 
compact support. We introduce a distribution 

J^,a{fn<^ fm) {p{fn)(pTr, p{f7n)Va)B';' {(p^,(pa) 

where the sum is taken over orthonormal bases of At^ and Aa- Here we use a model of ly' as (4.2). 
As usual, we associate to /„ (8) /m a kernel function on (U„(fc')\U„(A') x Um(fc')\Um(A')) : 

iCumMi.9'2:9'3^9'd^ E Mg'f'Cg's) E fM-'eg'd (4.3) 

C'6U„(fc') ?'eu,„(fc') 

and consider the following distribution: 

JUn f,n) /C/„8/„ (£(/i'i), K(/^i); eih'^),K{h',)) iy'ih[-'h'^)dh[dh',. (4.4) 

(H'(fc')\ff'(A'))' 

The above integral is not absolutely convergent in general and need regularization. In order to see what 
we should expect for these distributions, We avoid this problem in this paper. Plug in (4.3), we have 



(4-4) = // E fnie{h[r\'e{h',)) E fMKy'e<iA)y{h'f'h',)dh[dh', 

h[ h'^ C'6U„(fc') «'eu,„(fe') 
II E /«(£(/i'l)-'e'C'e(/^^))/™('^(/i'l)-'e'«(/i'2))^'(/i'^'/i2)d/i'ld/^'2 



KM' «'eu„(fc')C'eu„(fc') 



E E /„(£(C'/i'i)-^C'£(/i2))/™(«(r/i'i)"'«(/i2))^'(/i'f'/i2)d/i'id/i'2 

«'eH'(fc') C'ec/;.,„+i(fc')\u„(fc') 

= If E /„(£(/i'i)-^C'£(/i2))/™(«(/^'i)^'«(/i2))^'(/i'f'/i2)d/i'id/i^. 

H'ik')\H'{A') H'{A') C'eC/{..,„+i(fc')\U„(fc') 



(4.5) 



If we write h'^ = UiOl, then 

^'■^^^ / 11,,^, 

H'(k')\H'{A')U[r- „ + i(A') U„(A') C'e'^;'-,m + i(*-'')\Un(fe') 

/n ig'f ^Ikl ^ C'e (/l2 ) ) /m (^r ^ '« ('^2 ) ) V^li^'f ^ ) ^' (^^2 )d3i du'i d/l2 

-I 11.,^. 

H'{k')\H'{A'} U[r „ + i(A') U„(A') C'6'^;r,,„ + i(fe')\U„(fe') 

/n(5r'«^(/»2)"'lir'C'e(/l2))/rn(ffr')Wr')^'(/^2)d5;du'id/l^. (4.6) 

Define a function / e ■H(U„(A')) by 



/(5')= y /n(5'l5')/rn(5i)dgl. 

U^(A') 

Then 

(4.6)= E II /(^('^2)"'ii'r'c'e(/i;.))£(it'f')^'(/i2)d2i'id/i'2 
S / / 

C'e(C^;^™+i(fc')\U„(fe'))//ff'(fc')stab«'(fe')\-ff'(A')'^I'-,™ + i(A') 

/(K(/ii)-VfiC'£(/ii))V''(lir')i^'(/i2)dli'id/i^ (4.7) 
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where H' acts on U{r m+i\U„ by conjugation and [U{r m+i{^')\^n{k')) // H'{k') is the set of conjugacy 



classes of fc'-points. We introduce a fc'-algebraic group 

H' := H' X H' 

which acts on U„ in the foUowing way: for any fc'-algebra R, h' — h'(w']^, .9') ^ H'(i?) with £ 
?7{r g' e Um(i?) and 5 G U„(i?), we define the right action [g]h' := g'~^u.i^ g'u!29' ■ We also 

denote by [U„(fc')]/H'(A:') the set of /c'-orbits under this action. We define a character (also denoted by) 
V/ of H'(A') by 

£(h') =£(h'(^i;,ii^;g')) :=£(^r'^2)- 

Then 

(4.7)- E / /([C']h')VL(h')dh' 

E ^C'(/)-:^(/) (4-8) 

C'G[U„(fc')l/H'(fc') 

which is a decomposition of the distribution J according to the orbits. 

We denote by U„(fc')i.eg the set of all regular /c'-elements which will be defined in Section 4.3, Definition 
4.8. In particular, the H'-stabilizer Stab^ is trivial for C G U„(fc')i.cg by Proposition 4.11 and the 
corresponding term ©(/, C') Jc,'{f) is a weighted orbital integral. If / = ®v' fv' is factorizable, then 



where 



In summary. 



H', 



J{f) =JrM+JUf) 

E JcU) + JUf) 

C'e[U„(fc'),cg]/H'(fc') 

= E n 0{U,C)+J-ur{f). 

C'G[U„(fc'),cg]/H'(fe')i''GMfc, 



Now we discuss the relative trace formula on the product of two general linear groups. We iden- 
tify GLm^fc C GL„^fc with GL{W) C GL(F) and view GIjuM' C ReSfe/fe/(GL 

n,k) (res. G\jYa,k' C 

ReSfc/fc/ (GLm^fc)) through the basis ii^, wi, u>m, iuq, Wr, ui}. Let Z'^ (resp. Z'^) be the center 

of Gljn,k' (resp. Gl^m.k')- We denote ip = ip' o Tr. Let 11 and S be as above which are cuspidal. Since 11 
and E are the standard base change of representations of unitary groups, we need to introduce a period 
integral to single out such representations. 

Until the end of this section, we assume that n is odd, hence m is even. Since the other case is similar 
and will lead to the same fundamental lemma, we omit it in the following discussion. As pointed out in 
[F191], [F192] and [GJROl], the central character of 11 (resp. S) should be trivial on A' and 11 (resp. 
E (8)77) should be distinguished by GL„^fc' (resp. GLm.fc')- Hence we consider the following integrals as in 
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[GJROl] and [JR]: 

Z;(A')GL„(fc')\GL„(A') 

Vrnifs) = j <<fn(52)?7(detg2)d52 

^m(A')GL„(fc')\GL™(A') 

where we recall that 77 is the quadratic character associated to k/k' . 

Let Fn G 'H(GL„(A)) and Fm £ H(GL„i(A)). We introduce another distribution 

Jn,s(s;F„ (g)F„i) := ^B'^.,,.{s; p{Fn)^ii, p{F„,)tpY^)'Pn{m)Vm{.^Y^) 

where the sum is taken over orthonormal bases of An and Here, we take a model of v in the following 
way (be cautious that since we change the coordinates, the matrix form of the following element changes 
from (2.4)): 

v{}i) = V (ug) = V {h (n, n* , b; u, u*; g)) = ip{u) 

where 

■0(u) = (uj 2 + • • ■ + Ur-l,r + 'T-r.O + /^0"-0,r + W*,r-1 + ' ' ' + "2,1) (4-9) 

We associate to F„ ® F„i a kernel function on (GL„(/s)\GL„(A) x GL,„(fc)\GL„i(A))^ (averaged by 

Z'^(k')\Z'„{h') <6GL„(A;) Z'^(k')\Z'^(k') «eGL„(fc) 

and consider the following distribution: 
J(s;i^„ ® F„) 

- I I I 

^™(A')GL™(fc')\GL,„(A') (A')GL„(fc')\GL„(A') H{k)\H{k) 

K^F„(SF^{£{h),K{h);gi,g2)v{h )|det/i|^ ^ 77(det 52)dMgidg2 

= / / / / / 

•Z™(A')GL„(fc')\GL„(A') Z;(A')GL„(fc')\GL„(A') Z'^{k')\Z^{A') Z'^(k')\Z'^{K') H{k)\H{K) 

Y Fn{e{hy^ ziCgi) ^ i^m(K(/i)"^Z2Cg2)i'(/i~^)| det/i|^~^?7(det.g2)d/idzidz2dgid52 

C6GL„(fc) ?eGL,„(A:) 



GL„(A:')\GL„,(A') GL„ (fc')\GL„ (A') !7ir_„ + i,ir (fe)\H(A) 

^ F„(e(/i)-iC3i)-F^rn('t(/i)"\92)i'(/i"')|det/i|;"^77(det52)dM.gid52. (4.10) 

CGGL„(fc) 

We decompose h — ug and notice that the group H{K) is unimodular. Moreover, we change a variable 
9 ^ 92^9- Then 



<"°>^// / / S 

ff2 31 ;7ir.„ + i,i,-(fe)\C/ir,„ + i.ir(A) GL,„(A) CeGL„(fc) 



Fn{9~'92 2l-X9i)Fm{9~')Hu'n\dctg\-j, ^IdctgaU ^^Kdct 52)d5dMdgid52. (4.11) 
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Define a function Fg on GL„(A), which is holomorphic in s, by 

Fs{~g)= J K{g''g)F,„,{g-')\detg\l-hg. 

GL„(A) 

We also introduce the symmetric space S„ C KeSk/k' (GL„.fe) defined by the equation ss"^ = 1„, hence 

S„(fc') = {seGL„(fc) |.ss^ = l„}. 

We have an isomorphism GL„ fe/GL„ k' — S„ given hy g ^ Qff'^^- Define a finear map a : ?^(GL„(A)) 
7^(S„(A'))by 

ia{F)) igg^'-') ^ J F{g~g)d~g (4.12) 

GL„(A') 

and let Fs = (t{Fs). 

Combining these two operations together, we get 

(4.11)= j J F,ig^'u-'Cu^92)tiu-')\dctg2\l~^v{detg2)dudg2 

C6S„(fc')GL„(fc')\GL„(A') C/ir,„+i_i.(fc)\C/ir-,„ + i,i.(A) 

(4.13) 

Similar to the unitary case, we introduce the following fc'-algebraic group 

H := ReSk/k' {Ui^ xi GL,„_fc' 

which acts on S„ in the following way: for any fc'-algebra i?, h = h(u; g) with u G J7ir „i+i ir (i? O /c), g G 
GLm{R) and s 6 S„(i?), we define a right action [s]h g~^u~^su^g- We also denote by [S„(fc')]/H(fc') 
the set of fc'-orbits under this action. We define a character (also denoted by ip) of H(A') by 

and deth det g. Then 

(4.13)= I ^^.([C]h)^(h)|deth|r^Kdeth)dh 

CG[S„(fc')l/H(fc')H(AO/Stabf(fc') 

Y Jc{s;F,)^:J{s;F,). (4.14) 
Ce[s,.(fe')]/H(fc') 

We denote by S„(fc')rcg the set of all regular fc'-elements which will be defined in Section 4.3, Defi- 
nition 4.8. In particular, the H-stabilizer Stab^ is trivial for ( G S„(fc')reg by Proposition 4.11 and the 
corresponding term 0{s; Fg, () '■= Jc{s] Fs) is a weighted orbital integral. If Fg = ®v'Fs^v' is factorizable, 
then 



0{s-F,X)= n 0{s-Fs,,,X) 



where 



0(s;F,,,,,C) = j F,,,, ([C]h„OV^^,(h.' )??..' (dcth^O I det h„<, Mh„,. 

In summary, 

J(s; Fs) ^Jrcgis; Fs) + Jirr(s; Fs) 

Y Jds;Fs)+Jirris,Fs) 

Ce[s„(fc').og]/H(fc') 
= E n 0{s;Fs^,,X) + Jn.{s;Fs). 

CG[S„(/c')rog]/H(fe')i''6Mj^, 
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When s = i , the terms involving | det | disappear and we discard s in ah notations in this case. In 
particular, 



JciF) = OiF,C)= J F([C]hMh)7y(deth)dh 



H(A') 

which should be compared with 

Jc'(/) = 0(/,C')= / /([C']h')£(h')dh' 

H'(A') 

assuming that C and C are both regular. 

Remark 4.3. If the original functions Fn = ®vFn,v and F^ = ®vFm,v are factorizable, then Fs = ®vFs,v 
and Fs — ®v'Fs^v' are also factorizable. If for some (finite) place v\ Fm,v' has the property that 
{| detail,/ I Fm,yi{g) 7^ 0} is a singleton, then Fsy = Fyi are independent of s. In particular, this is 
the case for almost all v' . 

We will introduce the notion of matching orbits in the next section. It is expected that we have 
enough pairs (F, /) such that ii ( ■H- (' are regular and match, then 

0(F,C)-0(/,C')- 

The above relation should also be true place by place. In particular, let us consider the case where 
v' e Mfe/ splits into two places w,,Wo S M^. Then we may identify Sn,v' with the set of pairs {g»,go) G 
GL„^„, X GL„^-„^ with g^go = In, hence with GL„^^/ by ((?»,5o) >— >■ g»- Then Fyi becomes a function on 
GLn^yi and 

O{Fy,,0^ J JJ Fy,{g-'u~\u,g)f{u-'u,)du,du,dg 

for the generic character 

V^'(w) = V'' 0' {ui,2 H 1- Wr-l,r + '^r,0 + /3o«5,r + K,r-1 H ^ '"2,1)) (4-15) 

where j = (j, - j). 

On the other hand, we may identify \Jn,v' with the pairs {g»,go) such that go — w^} ^g^^wp'^, and 
with GL„ yi. Here 



Wr 



Wr 



where /3' = {(3',,P'o)- Then /„/ becomes a function on GL„_„/ and 

0(/.',C')- / // fv'{g'-'y^,-\'y^og')gK-'y^oW,dy!odg'. 

Moreover, in this case, that ( and (' match exactly means that C = C 6 GLny . Hence for 

fv [g) = Fy, (g) = J Fy, {g~g)Fy^ {g)A~g, 

we have 

0{Fy,X)^0{fy.X) (4.16) 

for all C regular. 



31 



4.3 Matching of orbits and functions 

Let k' be a number field or a local field and k/k' possibly split. We say two elements /3i, f32 G Herm(fc/fc')^ 
are similar, denoted by /3i ~ /32 if G GL,„(fc) such that (32 = ^g"^ Pig. We denote [Herm(fc/fc')^] the 
set of similarity classes. We write W = , V = if the representing matrix of W is in the class /3, 
and (resp. U^, H'^) for U(VK'^) (resp. U(T/''), H'). We define 

e(/3)=,7((-l)'^^det/3) e {±1} 

to be the e-factor. 

We first define the notion of pre- regular orbits. Recall that we have the action of H (resp. H'), and 
hence its maximal unipotent subgroup KeSk/kiUir „i+i,i'' (resp. (U[r m+i) )j '^^ (resp. U^). 

Definition 4.4. An element C S S„(fc') (resp. G U(^(A:')) is called pre-regular if its stabilizer under 
the action of Resj,/i,/t/ir j^r (resp. {U{r m+i) ) trivial. 

Let us start with the symmetric space S„ . Let B be the Borel subgroup of GL„ consisting of upper- 
triangular matrices and A = (GLi)" be the maximal torus consisting of diagonal matrices. Let W„ be 
the Weyl group of GL„ which is isomorphic to the group of n-permutations. We identity elements in W„ 
with permutation matrices, and hence identity W„ with a subgroup of GL„(A:). Moreover, let C W„ 
be the subgroup consisting of elements whose square is 1„. Let P be a standard parabolic subgroup of 
GL„.fe whose unipotent radical is U. We also choose a standard Levi subgroup M of it consisting of 
matrices with diagonal blocks. The group ReSfe/fc/P acts on S„ from right by [s]p = p^^sp^. First, we 
have the following lemma. 

Lemma 4.5. An element C G S„(A:') has trivial stabilizer under the action o/U(fc) C P(fc) */ and only 
if its orbit intersects [w]M(fc), where w = Wn is the longest element in W„. Moreover, the intersection 
contains at most 1 element. 

Proof. By [F192, Proposition 3], we have a Bruhat decomposition for S„(fc'): 

Sn{k')= ]l NB(fc). 

It implies that for general P, we have 

Sn(fc')= U MP(fc)= U MM(fc)U(A:). 

Hence in any U(fc)-orbit, there is a representative of the form [w]m. We assume that ^ = [w]m = 
m~^wm^ . Then its stabilizer is trivial if and only if 

{u-^wu^ =w I uGU(fc)} =={!„}. (4.17) 

But u^^wu^ = w is equivalent to wuw ~ , hence if w = w is the longest Weyl element, it will force 
u — 1„, and the [w]m is the only point where its orbit and [i/7]M(fc) intersect. 

Conversely, we need to show that if (4.17) holds, then w G [wJWm, where Wm C W„ n M(fc) is the 
Weyl group of M. We observe that any w G is a disjoint union of transpositions. We use induction 
on n. The case n = 1 is trivial and assume this for < n. If the transposition (1, n) appears in w, then we 
reduce to the case of n — 2 and we are done. Otherwise (1, a) will appear in w with 1 < a < n. Suppose 
that M — GL„j X • • • X GL„j (arranged from upper- left to lower-right) with n = rii + • ■ • + rit, > 
and t > \ (otherwise, it is trivial). If n — a < n^, then w' = (a,7i) is an element in Wm C M(fc). The 
conjugation w'~^ww' G will contain the transportation (1, n) and we are done. Otherwise, n — a > nt 
and consider the transportation {b,n) in w' with 1 < b < n. If — 1 < ni, then we can conjugate w 
by (1,6) G Wm and we are again done. The rest case is that b — 1 > ni. Then we define an element 
u G U(fc) whose entries are 1 at diagonals and positions (1,6), (a,n), elsewhere. Then wuw = u = u'^ 
which contradicts (4.17). 

□ 
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Applying the above lemma to P = Pir^_|_i ^t- stabilizing the flag C {vi} C ■ ■ ■ C X C X(BW(BE C 
X(BW®E(B {vr } C • • • C V^, which is standard according to our basis {vi, ...,Vr,wi, Wm , , Ur , ■ • • , ^"1 } 
used in this Chapter. Since w normalizes Pir ,„+i j^r, the C/ir „j+i ir(fc)-orbit of a pre-regular element C 
must contain a unique element of the form 



Pr(C) 



triO 



(4.18) 



^i(C)^-^ 

with ti{() G and Pr(C) G Sm+i{k')- Wc call it the normal form of 

Now we consider the unitary group \jf^. We fix a minimal parabolic subgroup Pg such that its 
maximal unipotent subgroup Ug contains U[t m+i- Let Ag be a maximal torus inside Pq and the 
Weyl group. Let P' be a general standard parabolic subgroup of whose maximal unipotent subgroup 
is U' and M' D Ag a Levi subgroup. The group (P')^ acts on from right by [g]{pi,P2) = Pi^gP2- 
We have the following lemma similar to the case of symmetric spaces. 

Lemma 4.6. An element ^' € U^(fc') has trivial stabilizer under the action o/ (U'(fc'))^ C (P'(A;'))^ if 
and only if its orbit intersects [w^] (M'(fc'))^ = M'(A:')w'^]V['(fc'), where is the longest element in 
W^. Moreover, the intersection contains at most one element. 

Proof. Wc have the usual Bruhat decomposition: 



P'{k')wP'ik') = Y[ \J'{k')M'{k')wM'{k')V'{k'). 



i«ew^,,\w,'?/w^ 



u.ew„,\wg/w^ 



Hence in any (U'(fc'))^-orbit, there is a representative of the form miwm2. We assume that (' = miwm2. 
Then its stabilizer is trivial if and only if 



u;U'(fc')w~^ nU'(fc') = {!„}. 



(4.19) 



Let i?+(Ag,U^) (rcsp. i?+(AQ , M')) be the set of positive roots of Ag (resp. in M'), then any double 
coset of WmAWtc/'^m' has a unique representative w satisfying w{a) < and w~^{a) < for all 
a G i?+(Ag , M'). Assuming that w satisfies (4.19) and the above condition, then w{a) < for any 
a e i?+(A(^,Ug). Hence w = w^. Conversely, if w ~ w'^. then (4.19) holds and the intersection is a 
singleton. 

□ 

Applying the above lemma to P' = m+i^ standard parabolic subgroup stabilizing the flag 
C {vi} C ■ ■ ■ C X C V. Since normalizes -Pfr the „j^i (A:')) ^-orbit of a pre-regular 

element must contain a unique element of the form 

ii(C^) 



Pr(C^) 



(4.20) 



with t,{C^) ek"" and Pr(C'^) G uf„+i(fc'), where Uf^+i = V{Wf^ ®E). We caU it the normal form of ( 
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Remark 4.7. There is a more intrinsic way to define the invariants ti. For C £ M„(fc), for i = 1, r, let 
(^[i] be the matrix leaving the first i eolumns and the last i rows of C and Si(C) = det (^[i] which is invariant 
under the action ( H> uCu' for u,u' e [/ir ,„_|.i ir(fc). Then C £ S„(fc') (resp. C,^ £ U^(fc')) is pre-regular 
if and only if s,;(C) G fc^ (resp. Si(C'^) G fc^), where we view C, (resp. C^) as elements in M„(fc) through 
the natural inclusion S„ C KeSk/ k'^n.k = End(T^) and C Kes^/ k'^n,k- Moreover, the invariants ti 
and Si are related by ti{() = Si-i{C,y Si{(y'^^ (so = 1) and similar for . 

Now we are at the point to introduce regular orbits. We have natural inclusions S,„+i C Kes/^/i^iyirn+i.k = 
End{W © E) and U^+i C RcSk/k'M^+uk- 

Definition 4.8. An element ^ G M,„+i(fc) is called regular if it satisfies: 

• ^ is regular semisimple as an element of M,„_|_i(fc); 

• the vectors {wq, £.wo, C^wq} span W (B E; 

• the vectors {w^ ,w'^£„ ...,w^£,"'} span ® £;^. 

An element ( e S„(fc') (resp. S U^(fc')) is called regular, if it is pre-regular and the uniquely deter- 
mined element Pr(C) G M„+i(/c) (resp. Pr(C'^) G U^+i(fc)) is regular. An H-orbit C G [S„(fc')]/H(fc') 
(resp. H'^-orbit G [U^(fc')]/H^(fc')) is called regular if any, hence all elements inside it are regular. 
Wc denote by M,„+i(A:)rog, GL™+i(fc)i.og := M™+i(A:)rog n GL„+i(A:), S„(A:')rcg (resp. [S„(fc')rcg]/H(fc')) 
and U^(fc')rcg (resp. [U^(fc')rcg]/H^(fc')) the set of regular elements (resp. orbits). 



To proceed, we need to recall some structural results in [RS07, Section 6], see also [JR], [Yun09] 
and [Zh]. To include the whole action of H (resp. H^), we need to consider the conjugation action (from 
right) of GLm.fe' (resp. U(^). We consider more generally the conjugation action of GL^.k- Recall that 
by our choice of coordinates, the group GLm,k embeds into Ghm+i,k via 



9 ^ 
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Let a,(0 = Tr (^A' C j for 1 < i < m+1 and b,{^) = w^Cwq for 1 < i < m. Let = det [(w^f -i)(u;j_i)]™:^\, 

Dt the matrix \{wq £^''^^){^^^^wq)]"^^^ and At = detDe. It is clear that if ^ is regular, then At ^ 0. 
Moreovcr, we have 



Lemma 4.9. Two regular elements ^ and ^' are conjugate under GL„j(fc) if and only if ai{^) = ai(^') 
and bi[^) = bi{$,'). The Ghm- stabilizer of a regular element is trivial. 

Proof. See [RS07, Proposition 6.2 & Theorem 6.1] for the (equivalent version of the) first and second 
statements, respectively. 

□ 

To include all unitary groups at the same time, we consider the set ilm+i of pairs (/3,^'^) where 
/3 G Her™(fc/fc')^ and G uf,+i(A:')rog := {C^ G M™+i(fc)reg | * {^^Y ^'^^ = /?'}, where is defined 
in (4.1). The group GL„(fc) acts on it™+i by (/3,C^)ff = ( 'g' Pg, g'^^/^ g). For f G S„,+i(fc')rcg := 
Sm+i{k') n Mm+i{k)ieg, WC deuotc by ^ (/3, ^'') if there exists g G GL,„(fc) such that ^ = g^^S,^g. The 
following lemma is considered in [JR] and [Zli]. 

Lemma 4.10. For ^ G Sm-i-i(fc')reg; there exists a pair (/3,^^), unique up to the Gljm{k)- action, such 
that ^ (/3,^^). Conversely, for every pair {/3,£,^) G ilm+i, there exists an element ^ G Sm+i{k')reg, 
unique up to the Gljrn{k') -conjugation, such that ^ <^ (/3,^''). 

Proof. We first point out that two elements G S„i+i(fc')rcg are conjugate under GL„i(fc) if and only 
if they are conjugate under GL,„(fc'). Assume g~^£,g ~ £,' , then g~^^g = g^'~^£,^'~^g^ = g^'~^£,9^ i hence 
9 = 9^- 
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The following proof is due to Zhang [Zh]. It is easy too see that for ^ £ Mm+i(fc)i.cg, £, and have 
the same invariants ai,bi. Hence by the above lemma, there is a unique element g S GL„i(fc) such that 

Hence we have g'^ = g. But we also have 



g-^^g = ^ e S,„+i(A:')rcg, then e S,„+i(A:'), 

t 



which implies that g = 'g. Together, we have g € lleim{k/k')^ . Moreover, '^'^(g 

^evCiik'Us and (5-1,0. 

Conversely, given any (/3,0 G -^m+i, then 



T,-l t 



-r,-l 



Moreover, there is a 7 G GL,„(fc) such that 7 ^^7 = We have (5'^' ^7 i'^7/3^ = i,"^' ^, i.e, 

(7/3"^)""^ ^ (7/3'^) = ^"^'-1. By regularity, 7/?'^ G S„i(fc'). Hence there exists g G GL,„(fc) such that 
7/3"^ = gg^'^^. Then 



i.e., g G Sm+i(fc')iog. The uniqueness part is obvious. 

All these considerations lead to the following proposition: 
Proposition 4.11. (1). There is a natural bijection 



□ 



[S„(fc')rcg]/H(fc') 



N 



U [Uf:(fc')reg]/H^(fc'). 

0e[Hcr,„(fe/fc')^] 



IfNC^ = then we say that they matching and denote by ^ C,^ . 

(2). The set S„(fc')rcg (resp. U^(fc')iogy' is non-empty and Zariski open in S„ (resp. U^j. Moreover, 
the H- stabilizer (resp. -stabilizer) of regular C, (resp. ) is trivial. 

Proof. (1). Start with an element Q G S„(fc')iog and consider its normal form. We get r invariants 
tr(C) and an element Pr(C) G Sm+i(fc')rcg- By Lemma (4.10), there is a pair {(3,^^) such that 
^ = g^^£,^g. We fix /?, then ^'^ is uniquely determined up to the U^(fc')-conjugation. We define an 
element by 



triC) 



trier 



GU^(fc')r 



By construction, C defines a map 

N : [S„(fc')reg]/H(fc') 



u 

/3G[Hcr™(fe/fc') 



K(fc')reg]/H'5(fc') 



which is injective. The converse is similar. 

(2). The openness is due to the fact that the pre-regular elements in both cases correspond to the 
unique open cell in the Bruliat decomposition and the fact that the three conditions in Definition 4.8 are 
open. The non-emptiness is essentially exhibited in [JR, Section 3] combining with the exponential map. 
For the last part, we prove it for ^ G S„(fc')i.eg and the case for unitary groups is similar. We write ^ in 
its normal form. If g~^u~^(^u'^ g ~ C,, then u^^ g~^C,gu'[ = C, with ui = g^^ug G J7ir „i_|_i it-(A:). We have 
g~^C,g = C- 5~^Pr(C)5 — Pr((^) which implies that 5 = 1„ and hence u = 1„. 

□ 
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It is clear from the above discussion that the regular orbit ( S [S„(fc')rog]/H(fc') or [U^(fc')rcg]/H'^(fc') 
is determined by its invariants <i(C) {i = 1, ...,r), ai{() := ai(Pr{()) {i = 1, ...,m+l) and bi{() -.^ bi(Pr{()) 
{i ~ l,...,m), and C ^ if a-nd only if they have the same invariants. We also denote = Tpi.(^), 
Dc = Dp,.(c) and A^; = ■ 

Now let k' be a local field and n odd. As suggested by the case where k/k' is split. We are going to 
formulate the conjecture on the matching of functions. First, let us define a suitable "transfer factor" t. 
Recall that we have a character 77 : k'^ — > C^. We define, for C G [S„(fc')iog]/H(fc'), 

t(C)-7y(Tc-(detPr(C))-^) 

which makes sense since m is even and • (detPr(C)) ~ G fc'^. 



Conjecture 4.12 (Smooth matching). Let k' he as above. Given any smooth, compactly supported 
function F G 7i{Sn{k')), there exist functions {f^)p for each /3 G [Ileim{k/k')^] such that 

o(F,c)=t(c)o(/^c'') 

for all C G [S„(fc')reg]/H(fc') and ^ ^ C^. Conversely, given any functions f^ G 'H{U^{k')) for each 
/3 G [Heimik/k')^], there exists a function F G H(S„(A:')) such that the above identity holds. We say that 
such F and {f^)i3 match and denote by F ^ {f^)i3- 



Corollary 4.13 (of Section 4.2). If k/k' is split, then the conjecture of smoothing matching holds. 



4.4 The fundamental lemma 

As usual, to establish the equality between two relative trace formulae, one need to prove the correspond- 
ing fundamental lemma. We now formulate our fundamental lemma. We allow m to be any nonnegative 
integer. 

Let k' be a non-archimedean local field and k/k' a separable quadratic field extension. There are 
only two non-isomorphic hermitian spaces of dimension m > over k which is distinguished by the factor 
e(/3). We will use the superscript ± instead of /? for e(/3) = ±1 in the following notations. 

We now assume that k/k' is an unramified field extension and ip' : k' is an unramified character. 

As before, we write 0' (resp. 0) the ring of integers of k' (resp. fc). We denote by val : fc^ ^ Z the 
valuation map. We also assume that /3o G 0'. Then will have a self-dual o-lattice Lw which extends 
to a self-dual o-lattice Ly of . The unitary group (resp. U^) is unramified and has a model over 
0'. The group of o'-points U^(o') (resp. U^(o')) is a hyperspecial maximal subgroup of U^j(fc') (resp. 
\J^{k')). Wc also identity GL„(o) with GL„(iv')- a- hyperspecial maximal subgroup of GL„(A:) and let 
S„(o') :=S„(fc')nGL„(o). 

Wc denote by ■H(U+(fc')//U+(o')) (resp. ?^(GL„(fc)//GL„(o))) the spherical Hecke algebra of U+ 
(resp. GL„_fc). There is a base change map b : ?^(GL„(A:)//GL„(o)) H(U+(A:')//U+(o')) and rccaU 
that we have a linear map a : 'H(GL„(A:)//GL„(o)) — >■ 'H(S„(A:')) similarly defined as (4.12) for the local 
case. Moreover, we define 

^^^^^ r(-l)-(TcnL..(C)) ™i,,dd; 
W_-[^^vai(Tj TO is even. 

Conjecture 4.14 (The fundamental lemma). For any element F G 7{(GL„(fc)//GL„(o)), the func- 
tions F = <y{F) and (/^, /^) match, where = HF) '^'^'^ .f^ = 0- 
In particular, we have 



O(ls„(0')>C) 



lt(C)0(iu+(o')'C+) C^C+eU+(fc'); 
10 C^reu-(fc') 



36 



where 



0(ls„(o'):C)= J ls„(p')([C]hMhMdcth)dh; 

H(fe') 



H+(fc') 

It is easy to see that C f^"*" 6 U^(fc') if and only if val (A^;) is even. 

Proposition 4.15. If val (A^) is odd, then 

0(ls„(oo,C)=0. 
Proof. The following is a modification of an argument in [Zh]. Let 

Wr 



W 



lm+1 

Wr 



then it is easy to see that ls„(o')(s) — ls„(o') *'^''^)- write in its normal form (4.18), then 

0(ls„(p')>C) 

ls,.(o') (w V 's'C'ff"^ *u~^w) 7p{ur^)r]{det g)dgdu 



C/i-.m+i.iKfc) GL„(fc') 

= J J ls„(o') ((^ V'"'^^)"' '5(i«'H VM«^ V^(w"')'7(det5)dffdii. 

(4.22) 

But w *'Cw and ^ are both of normal form and have the same invariants. In the proof of Lemma 4.10, 
we see that there exists h 6 GLm{k') such that w *^(w — h~^C,h and rjideth) = —1. Moreover, V'd*) = 
ip{w ^u^'~^w). After changing variables w *uj''~^w ^ u, h ^g~^ i— >■ we have 

(4.22) = - J J lsUo'){u-'9-\9v:)t{u-')videtg)dgdu 

which implies that 0{ls^{o')X) — 0- 

□ 

5 A relative trace formula for U„ x U^: Fourier-Jacobi periods 
5.1 Fourier-Jacobi models and periods 

Let us briefly recall the definition of Fourier-Jacobi models and periods for unitary groups in [GGP]. 
First, let us consider the local situation, hence k' is a local field. Let ^ be a hermitian space over 
k of dimension n with the hermitian form ( — , — ) and W C V a subspaee of dimension m such that 
the restricted hermitian form (— , — )|w is non-degenerate. We assume that the orthogonal complement 
= X (B X* such that X, X* are isotropic and of dimension r. Hence n = m + 2r. The hermitian 
form restricted on W (resp. X ® X*) identifies W (resp. X*) with W = (resp. X). We denote 
V{V) (resp. U(M^)) the unitary group of V (resp. W) which is a reductive group over k'. Let P^m be 
the parabolic subgroup of V(V) stabilizing X and its maximal unipotent subgroup. Then J7^„ fits 
into the following exact sequence: 

^ A' ^ ^ Homfe(l^, X)^0. 
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Let £'-^ : X —i' k he any nontrivial homomorphism (if exists) and let U'x be a maximal unipotent 
subgroup of GL{X) stabilizing i'-^. The homomorphism induces a homomorphism A^^Sc ■ At ^ ~^ 
At k — ^' a homomorphism 



Rcsfc/fe,^:^ : Hom;,(M^,X) Homfc(M^,fc) = Honifc, {^cSk/k'W,k') = {Rcsu/u'WY 

The hermitian pair ( — , — ) oiW induces a symplectic pair Tr (— , — ) on the 2m-dimcnsional fc'- vector space 
RcSfc/fc/W^. Hence (Resj;/fc/ VF)^ is identified with ReSi-yfe/VK through this pair. Again, let H (Res^^/j,/ VF) 
be the Heisenberg group, then we have the following commutative diagram: 



■ U' 

^ r.m 



Res 



0- 



H(Res 



k/k 



w) 



■ RcSk/k'W 



(5.1) 



■0 



For a nontrivial character : fc' ^> , we have a Weil representation ui'^, of H (Kes^/k' W) Mp (Resj, / j,/ W) . 
If we choose a character : — > such that yul^'x — t], we will have a splitting map 



Mp(Resfc/fc,W^) (5.2) 




U(M/)C ^Sp {Rcs,/,,W) 

(cf. [HKS96, Section 1,2]). By restriction, we get a Weil representation w^,, of H (Res^/fc/W^) x \J{W), 
and hence a representation of U^„^ x: U(M^) through the middle vertical map in (5.1). Let A' : U'x — > 
be a generic character. Then we define v'^,, ^ = uj'^, ^® \' which is a smooth representation of H' := 
^rm ^ i^'x ^ U(M^)). As before, we have an embedding H' ^ U(F) x U(W^). Then up to conjugation 
by the normalizer of H' in V{V) x U(W^), f^,, is determined by modulo Nmfc^ and /i. 

Let TT (resp. cr) be an irreducible admissible representation of U{V) (resp. U(M^)). A nontrivial 

element in Hotiih' ^tt (g) cr (g) u'^, ^,C^^ is called a Fourier- J acobi model of tt cr. In particular, when 

k/k' is split, then the Fourier- Jacobi model is just the (r, r)-Fourier-Jacobi model for general linear groups 
introduced in Section 3.1. We have the following multiplicity one result. 

Theorem 5.1. Let k be a non-archimedean local field of characteristic zero and tt, cr as above. Then 
dimcHoniff/ {^tt ® a ® v'^,^^, C^'j < 1. 

Proof See [GGP, Section 14]. 

□ 

Now, we discuss the global case. Let k/k' he a quadratic extension of number fields, tp' : k'\A' — > 
nontrivial, fj, : k^\A^ — > such that /iU'x = ^iid A' : U'x{k)\U'x{A.) C""^ a generic character. Then 
we have the pair {H' , v'^,, ^) in the global situation. To define a global period, we need to fix a model for 

the Weil representation. Let L C (RcSfe/fc/iy)^ be a Lagrangian subspace. Let 5(L(A')) be the space of 
Bruhat-Schwartz functions on L(A'). 

For (j) G 5(L(A')), we define the theta series to be 

0^',x'Ah',cl>)= E A'(/i') K'.^(/^')0) H 

w£'L{k') 

which is an automorphic form on H' . 

Let TT (resp. cr) be an irreducible tempered representation of U(V^)(A') (resp. \J{W){A')) which occurs 
with multiplicity one in the space Ao{U{V)) (resp. Ao{V{W))). We denote by A-,r (resp. Aa) the unique 
irreducible tt (resp. cr)-isotypic subspace in Ao(U{V)) (resp. Ao{U{W))). 
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Definition 5.2. The following absolutely convergent integral is called a Fourier- J acobi period of tt cr 
(for a pair {H' ,v'^, ^^)): 

JUy-^{ip^,^a;4')-= j v^{e{h'))^„{K{h'))e^j,^^_,{h'-4>)dh', eX,<^a e^.,<^e5(L(A')) 

H'(k')\H'(A') 

where dh' is the Tamagawa measure on H'{A'). If there exist (/j^, ipa, (j) such that JiJr "' '^ [fTr, fa',4') Oj 
then we say n (E) a has a nontrivial Fourier- Jacobi period. 

It is obvious that JlTt'*' '' defines an element in 

Hom^,(A,) (tt (T «) i^^,^^, = Hom^^^ (n^' ® cr„' (g) v'^-^^^^, , . 

■u'eM^,/ 



We choose a basis {fi,...,Vr} or X as in Section 4.1. We denote by {vi,...,Vr} the dual basis of 
X. We also choose a basis {wi, ...,w„i} of VF. Let /3 — [{wi,Wj)]'^^^-^ G Her„i(fc/fc')^. We identify 
U(F) (resp. U(VF)) with a unitary group of n (resp. m) variables U„ (resp. U„i) under the basis 
{vi, ...jVrjWi, Wm , , • • • , wi } and view U„i as a subgroup of U„ . Let C/{ r „ = „ xi U'x be a unipotent 
subgroup of U„. Then the image of H'{A') in U„(A') consists of the matrices h' = h'{n' ,b';u';g') = 
u'{n' ,b';u') ■ g' where g' S Uj„(A'). Here, 



u' = v^{n',b';u') = 



where n' e Mr,„i(A), 6' G Her,T(A/A'), u' S ?7x(^); '^/s === ^/^"^ 'n'^w^ and u' = Wr "^w^. If r > 0, 
let be the maximal unipotent subgroup of the parabolic subgroup of \J{{vr,Vr} ffi W) stabilizing the 
flag C {vr}- Let = yi \J(W) (resp. = V{W)) if r > (resp. r = 0). Then there is a map 
H' — > H^. We write = w-l-g' to be the image of h' under this map. Then we have 



n 



Wr 



b' + 



G ,„(A') 



5.2 Decomposition of distributions 

This time, we start from the relative trace formula on general linear groups. We identify Gljm,k C GL„^fc 
with GL(M^) C G\j{y) and view GL„^fc/ C Resj,/fc/ (GL„^fc) (res. GL^^fc/ C ReSfe/fc' (GLm^fc)) through the 
basis _ 

Recall that the Weil representation realizes on the space S{W'^ {A)) where ip ~ if;' o Tr. We let 

r r r r 

^♦=0fcyv. ^^^0fc-^v. p^*^0fc'^. VFO==0fc-«;, 

i—1 i—1 i—1 i—1 

which are vector spaces over k'. Then W = W* © W^, = VF* © W'^ . We also let = M^* © M^* 
be a vector space over fc'. We define a linear map from S{W'^ (A)) to iS(VF^(A')) by $ H> $'1^. where 

$t(w*,u;*)= y $(w*,u;'')V'(u''^(w*))dw'^ 
for $ G iS(Ty^(A)) and the self-dual measure on W^^(A'), which is an isomorphism. 
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If r > 0, let t/^ be the maximal unipotent subgroup of the parabolic subgroup of GL({iv, Vr} © W) 
stabilizing the flag C {vr} C {vr} © W. Let W ^ W x GL,„ (resp. W = GL,„) if r > (resp. r = 0). 
Then there is a map H — > . We write /i^ = u} g = u} (n*, n^, n*, n^, fe^) g to be the image of h under 
this map, where 



with n* e Mi^m{k'), rf e Mi,„(fc-), n* G M^,i(fc'), S M„,i(fc-). 
We define a representation of i/(A) on iS(VF^(A')) by 

4_(/^)a>t = (.^_(;^)$)^ 

It is easy to see that factors through W and 

fa;^_(/i^)$^) («;♦,«;*) = r;(dct g)^^ (6^ + w*n* + n'^w*) $^ ((m;* + n*)^, - n*)) (5.3) 
Moreover, we have the Poisson summation formula: 



1 n* + n 



Until the end of this section, we assume that n is odd, hence m is also odd. Since the other case is 
similar and will lead to the same fundamental lemma, we omit it in the following discussion. We proceed 
exactly as in Section 4.2 and take ^ to be the one used in (5.2) which is unitary. Let F„ e H(GL„(A)), 

G 'H(GL„(A)) and $ G <S(V1^^(A)). We introduce the following distribution 

Jl^{s; Fn ® F^; $) := ^ TJ^f- (s; p(F„)^n, K^^™)^'!]; $)^(^n)^(</?E) 

where the sum is taken over orthonormal bases of and A^. 

We associate to Fn ® Fm a kernel function /Cf„(8F„, (51, 52; ff3, 54) and consider the distribution 

J^(s;K0F™;$) = j II 

^m(A')GL„(fc')\GL„(A') Z;(A')GL„(/c')\GL„(A') H{k)\H{A) 

!CF„(8>F,A£{h),K{h);gi,g2)9^j-j,ih,<S>)\dcth\l'hhdgidg2. (5.4) 
Proceeding similarly as in (4.10), we have 

<^-^>=// / / S 

ff2 Si ;7ir,„,i'-('^)\C/i'-,™,i'-(A)GL„(A) CeGL,.(/c) 

-F'«(ff"^ff2'^^i"^Cffi)-P'm(g"^)%Ijf(M25,^')|det5|^"^|det gsir^d^dudgid^z 

^ / / / 

GL„(fc')\GL„(A') (7ir ,„ ir(fc)\;7ir.„^ir(A) GL,„(A) C6S„(fc') 

(^(^n)(5^^52"V^Cu^i725^)i^m(5~^)%,A,7i(M25,*)l detail"^ I det52ir^d<7dud(?2. (5.5) 
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Unfolding Uir „i.i'-ik), we have 



C6[S„(feO]/C/i^,™,i^(fc)GL„(fcO\GL„(A')stab,-i'-,™.i.-(,,)^y^^^^^(^)GL„(A) 

(^{K)i9'^g2\[C]u)929lFrn{g'^)0:^j^j,iM2g,'^)\dctg\l~^\dctg2\l~hgdu^^ 

E 



<^{Fn){g-H[92'C92]u)glF^{g-')9^j- (52, ^^^^M'J') V^d^"')! detgir Vetgsir'd^dffadu. 

(5.6) 



But we have 



w'lew^k') 



a:eMi_„(fe') yeM„_i(fc') 

To proceed, we introduce a fc'-variety 

As before, we let H ~ Rcs^./^/ ([/ir „j x"-) >^ Ghm.k' which acts on S„.m in the following way: for any 
fc'-algebra R, h = h{u,g) with u G J7i'-,m,i'-(-R ® k), g S GL,„(_R) and [s,x, y] G S„,m(-R), we define a 
right action [s,x, y]h = [g~^u~^ su'^ g, xg, g~^y]. We also define "(/"(h) = ''PiMT^) and det h = detg. Then 



'->^ S / / 



[f,x,y]e[S„.„(fe')]/H(fc')gtabH^ ^j(fc')\H(A') GL„(A) 

a(F„)(g-i[C]h5-)^^,„(.9-i) (c.^__(hg)$t) (x,yMh)| det h|7^ | det^irMgdh 

E ^[c,.,«](^;^"®^™;'J')- (5-7) 

[C,^,y]6[S„.™(/c')l/H(A;') 

We denote by S„.„i(fc')rog the set of all regular fc'-elements which will be defined in Section 5.3. In 
particular, the H-stabilizer Stab^ ^ y-^ is trivial for x, y] regular and the corresponding term 

0^(s; K ® F„; $, [C, X, y]) Jj^^^^ (s; F„ ® $) 

is a weighted orbital integral. If F„ = '^v'Fn.v', F,n = ®v'Fm,v' and $ = ®v'^v' are factorizable, then 

0^(s;F„ i^„,;$, [C,a;,y]) = J| O^^,, (s; F„,^,. (g) [C, x, y]) 

where the local orbital integrals are defined similarly as in (5.7). In summary, we have 
J^{s- Fn ® F„,; $) - j:^^,(s; K ® F,„; $) 
= E n C'/.„/(s;^«V ^^m.^'l^t-'JCa;,^])- 

[C,a:,y]e[S„,„(fc')rog]/H(fc')i''GMfc, 
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When ,s = i we discard s in all notations. In particular, 



'^i.M ® Fm -, $) = 0^(F„ ® F™; $, [C, X, y]) 

-- J J a(F,0(5-i[C]hg-)i^,„(5-i)(wJ_(hg)$t)(a:,y)V^(h)d3dh 



(5.8) 



H(A') GL„ 

when [<^, a;, y] is regular. 



Now we describe the relative trace formula for unitary groups. Let /„ G 'H(U„(A')). /,„ G 7^(Um(A')) 
and (pi e 5(L(A')) for i = 1,2. We introduce a distribution 



Jt'a'^ifn ® fra, <I>1 ® ^2) := ^ ^Jr'''-'' (pifn)^., piM^a] M^^Jr^' i^^,ipy,cP2) 

where the sum is taken over orthonormal bases of Ajr and Aa- As usual, we have a kernel function 

^fn^fm- Let 



{H'(k')\H'(A'}f 



{H'{k'}\H'{A'}y 

Collapsing the summation over ^' and changing variable <72^^5i ^ g'l, then 



(5.9) = 



// 



U„(fc')\U™(A')([7;^_(fe,)\[/;^ ^jA'))'U™(A') C'eU„(fc') 

fnig[~^g'2~^u'i~\'li^292)frn{g'{'^)0— jr j; (vfig2, i^^jjT j;ig'i)(pi) ^'^',A',m ("252' </'2) dg^ du'^ dw^ dg^ . 

(5.10) 

Recall that we define an fc'-algebraic group H' in Section 4.2 which acts on \Jn,m U„ x Res/j/^/Mi m^fc 
in the following way: for any /c'-algebra R, h' = h.' {u[,u^2, g') S H'(i?) and [g, z] G U„(i?) x Mi^m(^® k), 
we define the right action [g, z]h' = [5'~^M'i~^(?U2ff': ^ff']- We also define 

V/(h') =£(h'(u;,u^;.g')) := niLi-'vf2) 

and det h' = det g' . 

There is a map H' H}, where 

= iJ* X 

(resp. U„i) if r > (resp. r ~ 0). We denote by = h.^{ul,uh d') ^^c image of h' = h'(u'j, g') under 
the above map, where 



1 K - ^ 2-^ 

1 



for nj e W^^(A) and 6f £ 
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Lemma 5.3. Let be the tensor product ^^j^ ®^'ji>' ^ viewed as a smooth representation o/H', then 
it factors through H' — ^ . We have an intertwining isomorphism 

: {S(U^^'))f^ S ({Rc^k/k'WY (A')) = 5(1^^ (A)) 

such that: 

(1) For (jyeSiW^iK)), 



(5.11) 



which does not depend on ^, hence justifying the notation. 
(2) We have 

0^;y,j: (it'lff', 0l) 0^,,y,^ (U^ff', 02) = £(h') ^ (uji-ih') (</>! ® (z) 

where h' = h'(u']^,M2 7 5')- 

Proof. The isomorphism is by [HKS96, Proposition 2.2 (i), (ii)]. Actually one can construct an ex- 
plicit intertwining operator by a (partial) Fourier transform, which implies (1) and (2) by the Poisson 
summation formula. □ 

By the above lemma and repeating the process in (5.5), (5.6), (5.7), we have 



(5-10)= E / / 



fnig'-'mfrnig'-') (^^.(h') (c.^ _(,g')'/'l ® ^2) {z)£{h')dg' dh' 
= ■■ E J[f::^]{fn(E>fm;<Pl^h)- (5.12) 

[C',^]e[u„,„(fe')l/H'(fe') 

We denote by U„^m(fc')icg the set of all regular fc'-elcments which will be defined in Section 5.3. In 
particular, the H'-stabilizer Stabj^, is trivial for [(^' , z] regular and the corresponding term 

® /m; ® ^2, [C, Z]) := J((',;^]ifn ® fm] 01 ® ^2) 

is a weighted orbital integral. If /„ = ®v' fn,v', fm = ®v' fm,v' and 4>i ~ ®v'4>i,v' are factorizable, then 

CiA',M(/n ® /m;^! ® 02, [C',^]) = C'v;, (/".i-' ® fm,v'-Aiy ® (l)2.v' AC , z\) 

where the local orbital integrals are defined similarly as in (5.12). In summary, we have 

J'^''^(/n ® fm-, 01 ® 02) " J-^/^ifn ® fm', 01 ® 02) 

= E n '^V':,,M„'(/",i'' ^'^z,,,',^',^])- 

[C',^]6[u„.™(fc')rog]/H'(fe') v'eu^, 

In general, if = 0^*'' eg) 02*^ is a finite sum, then we simply define the distribution and the 
weighted orbital integral O as the sum of the corresponding terms defined above. 
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5.3 Matching of orbits and functions 

Wc first consider the following orbital problem which is a little bit different from the Jacquet-Rallis case, 
the one considered in Section 4.3. 

We define the affine space OTm M„i x Mi.„i x M,„,i. 
Definition 5.4. An element [^,a;, y] G 9Jt„i(fc) is called regular if it satisfies: 

• ^ is regular semisimple as an element of Mm(fc); 

• the vectors {x, x^, a;^™~^} span the fc-vector space Mi^m(fc); 

• the vectors {y, ^y, span the fc-vector space Mms{k)- 

Let ai{[^,x,y]) = Tr (A''?) for 1 < i < m, bi{[^,x,y]) = x^'y for < i < m — 1, 

X 



xC^-^ 

D[^ ,j, ,^] be the matrix \xC^-'~'^yY^-_-^ and A[^ ,j.,j,] = detD[^ It is clear that A[j .j, j,] 7^ if [5,a;, j/] is 

regular. The group GL„i acts on 9Jt,„ from right side by [S,,x,y]g = [9 '^S,g,xg,g~^y\ and Oj, 6i, 

and 3. J,] are all invariants under this action. We denote by Sflrn(fc)rcg the set of all regular elements. 

We have 

Lemma 5.5. Two regular elements [^,x,y\ and [^',a;',i/'] are in the same GL,„(fc)-or6ii if and only if 
they have the same invariants Oi and hi. The Ghm- stabilizer of a regular element is trivial. 

Proof. See [RS07, Proposition 6.2 & Theorem 6.1] for the first and second statements, respectively. 

□ 

We define two more spaces: 

&m = {[Lx,y] e »l™(fc)rcg I i e S,„(fc'),a: g Mi,,„(fc'),y g M,„,i(fc')}; 

= {[/?; ^, z*] I /3 G Her,„(fc/fc')^ [^^ ^, z*] G 9Jl,„(fc),.eg, ^ e U^(fc'), = 'zJ) 

where Uf„ = \](W^). ForilJ^, we also define a right GL,„(fc)-action by [I3;£_^,z, z*]g = [^g'^ (3g; g'^(,g, zg,g~^z*]. 
For G 6™, we denote by [^,x,y] ^ [f);^^ , z, z*] if there exists g G GL,„(fc) such that [^,x,y] = 

[^^ , z, z*]g. We have the following lemma which is similar to Lemma 4.10: 

Lemma 5.6. For [S,,x,y] G &m, there exists an element [P;^^ , z, z*] G it^, unique up to the GL,„(fc)- 
action, such that [S,,x,y] <^ [/3; z, z*]. Conversely, for any [/3;£^^ , z, z*] G itj^, there exists an element 
[£,,x,y] G &m, unique up to the GLm{k')-action, such that [£^,x,y] [/3;£^^ , z, z*]. 

Proof. We first point out that [^, x, y], [^', a;', y'] G &m are conjugate under GL„i(fc) if and only if they are 
conjugate under GLm(fc')- Assume that [^,x, y]g = , x' , y'], then g~^^g = C implies that g'^'~^^g'^ = C'; 
xg = x' implies that xg'^ = x'; g^^y = y' implies that g^'^^y = y', hence g = g'^ . 

It is easy to see that for \£,,x,y] G 9K(fc)reg, [^,a^,y] and [*^, *y, 'x] have the same invariants, hence 
there is a unique g G GLm(fc) such that g~^£,g = xg = g~^y = *x. Now if [^,x,y] G &m, then 
[*^, 'y, *x] G 6m, which implies that g = g^. Moreover, we have g = ^g and '^"^ (ff"^) C = .9^^- Hence 
g~^ G Hcrm(fc/fc')^ and ^ G Uf„ ^ (fc'). We also have y = (.9^^) ^ 'a;"^ which means that [g~^;^, x, y] G Uj^ 
and [^,x,y] <^ [jg"^^,x,y]. 

Conversely, given any [/3; ^, z, z*] G Since ^^P'^C = Z?"^, Z?"^'"^ ^C/?"^ = we have 

'z](3^ = [c^-\z\iz*y] 
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since z* = /3 ^ 'z"^. Moreover, since z,z*] and *z] have the same invariants, there exists 

7 G GLm(fc) such that z,z*]7 = [*^, hence 

which imphes that 7/3"^ S S„i(fc'), hence 7/?^ = 99^'^^ for some g e GL„j(fc). Then 

and zgg'^'^^ = z"^ implies zg = {zgY; g'^g^^z* = (z*)'' imphes g^^z* = [g^^z*^ . In all, [^,z,z*]g = 
[(7~^^(7, zg, g^^z*] G &m- The uniqueness is obvious. 

□ 

Recall that we denote [Herm(fc/fc')^] the set of similarity classes. We write W = , V = if 
the representing matrix of W is in the class /3, and (resp. U^, U^„, H'') for \]{WP) (resp. U(y^), 
\5{VP) X RcSfc/fc,Mi,™,fc, H'). We have 

Definition 5.7. An element [C,,x,y\ G S„_m(fc') (resp. [C^jz] G U^ ,„(fc')) is called pre-regular if the 
stabilizer of C (resp. C,^) under the action of ReSfc/fc't/ir_„^ir (resp. (C^i^^m)^) is trivial. 

Applying Lemma 4.5 to P = Pi>-,m,i'-, the /7ir-_m,i'-(fc)-orbit of C for which [C,a;,y] is pre-regular must 
contain a unique element of the form (4.18) with ti{C) G and Pr(C) G Sm(fc'). We call the triple 
[Pr(C),x,y] G 9Jlm(fc) the normal form of [C,,x,y\. 

Applying Lemma 4.6 to P' = Pf, the (C/{.^„)^-orbit of for which [C^,z] is pre-regular must 
contain a unique element of the form (4.20) with ti(C'^) G fc^ and Pr(^'^) G U^j(fc'). We call the quadruple 
[;3,Pr(C''), z,z*] G Her,„(fc/fc')'' x OT„(fc) the normal form of [C^,z]. 

Definition 5.8. An element y] G S„,„i(fc') (resp. [C^i-^] G U^„j(fc')) is called regular if it is pre- 
regular and its normal form [Pr(^),a;,y] G ©m (resp. [/3; Pr(C'^), z, z*] G il5„). We have the notions 
Sn,m(fc')rog, m(^')rog for the sets of regular elements. 

As before, we have the following proposition whose proof we omit. 
Proposition 5.9. (1). There is a natural bijection 

[S„,™(fc')rcg]/H(fc') ^ H [U^,„(fc')rog]/H^(fc')- 

^e[Hcr,„(fe/fc')x] 

//N[C^,z] = [C,,x,y], then we say that they match and denote by [(,x,y] O [C^i^]- 

(2). The set Sn,m{k')reg (resp. „(fc')i-cgy' is non-empty and Zariski open in Sn,m (resp. 
Moreover, the K-stabilizer (resp. -stabilizer) of regular [(^,x,y] (resp. [(^^,z]) is trivial. 

It is clear that the regular orbit [C,x,y] G [S„,„(fc')rcg]/H(fc') (resp. [C'^,z] G [U^^„,(fc')rcg]/H''(fc')) 
is determined by its invariants ti(C) (resp. ii(C^)) (* = l:---:^)i o,ii[C: x,y]) := ai([Pr(C), x, y]) (resp. 
a,([C^z]) := a,([Pr(C'5),z,z*])) {z = l,...,m) and h{[Cx,y]) &,([Pr(C), x, y]) (resp. b,{[C^,z]) := 
6i([Pr(C^), z, z*])) (i = 0, ...,m — 1), and y] O if and only if they have the same invariants. 

We also denote by T[c,x,y] = T[Pi{c).x,y], 'D[C,x,y] = 'D[Pr{Q)^x,y] and ^[c,x,y] = ^[Pr{C),x,y]- 

As at the end of Section 4.2, we would like to compare the weighted orbital integral O^^, {Fn^v' ^ 
Fmy,^v', K,x,y]) and , {fny ® fmy,4>iy ®4>2,v'^ [C'l A) when v' splits into two places v, and 

of k. 

As before, we identify S„_^' with GL„,i/ (resp. U„,u' with GL„ „/) and S„,m,u' with GLn^v' x Mi^^^^/ x 
Mm,i,„' (resp. Xin.m,v' with GL„,„' x {Mi,m,v'f)- Then [C,a;,y] o [C, (a;, ^yPo)] where ^ = (/3.,;5o). 



45 



By (5.3) and (5.8), wc have 

= J JJ JJ 'yiFn,v')i9,'g-\,\u,gg,)Frn,v.i9,')Frn,vA9o') 

«L''-'(C/i.„„,..,.OtGL„,„0' 
f ( \U( ^K)l+K)oA -l( «). + «)o 



2 y-"'" V 2 



y I I dg.dgodu,dit„dg (5.14) 
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On the other hand, by (5.11) and (5.12), we have 

t;'; 01,'u' ® 02,-u', [C, (a;, 'y/3o)]) 

y fn,v'{9,^9^^U.,^CUo9)f7n,v'{97^) 



(^(^W^_(g.)01,„') ® (/'2,,;') 



f (^i7'lio)V'' b- (for,r).-(&r,r)„+. 



2 / V 2 

2 + 2 



2/ d5.du,du„d5. (5.15) 



To compare (5.14) and (5.15), we need to invoke the origmal models of the Weil representations. In 
particular 

('^V^,7r(ff»'5o)*«') ix,y) = ti{detg,^go)\detg,go\^, J <i>y' {{x + z)g,,{x - z)9o)ilj'Uzy)dz 



M, 



where (/i,/i ^) with abuse of notation. If we identify L„/ with Mi^m.v, (possibly through a Fourier 
transform), we have 

(^(^a;^_(5,)(?!)i,t,'^ ® 02,1,') {x, ^y^^) = ^J■idct g,^)\dct g,\^, J (jii^yi {{x + z)g,) 4>2,v'ix - z)'ijj'{jzy)dz. 

M, „ „, 
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Now if we suppose that = <&, ^"0 is factorizable with respect to the two components. Let 
Fm,v, = fm,v', ^* = 4'i,v' and assume that the following function on {h,w) G GL„_^' x Mi^rn,v'- 

J (j{Fn,v'){hgo)F,n^y^{g~'^)^o{wgo)fJ.{det go)\det go\^,dgo 
is equal to J2t fnl' ® (1^2^' ' t^en 

0^:1,,, [C,(x, 'y/3o)]), 

■i 

hence the smooth matching (of functions) holds! 

Remark 5.10. As we see in the above calculation, there are several differences in the case of Fourier- Jacobi 
periods: 

• The data of test functions involve Schwartz functions on the groups as well as on the linear spaces; 

• Even for the split case, smooth matching is not obvious and we need to choose linear combination 
of functions to make corresponding weighted orbital integrals equal. 

For almost all split places v' where everything is unramified and the test functions are the characteristic 
functions on corresponding maximal compact subgroups or lattices, then the two orbital integrals are 
equal. 

Inspired by the split case, we conjecture that, similar to Conjecture 4.12, the smooth matching of 
functions holds for all places v' . We omit the explicit form of this conjecture in the current case. 



5.4 The fundamental lemma 

We now state the fundamental lemma for the Fourier- Jacobi periods. We use all the notations in the 
beginning of Section 4.4, except that we define 



'(-i)^^i(T[c,x,yi-nr=it.(c)) 



m IS even; 



t([C,^,2/])= '(T,, A . (5-16) 

[(-1) v-^K'^'fl'' m IS odd. 

For simplicity, we only consider the fundamental lemma for unit elements which is 

Conjecture 5.11 (The fundamental lemma). Assume that k/k' , ip' , ^ are unramified and j e 0. 
Then we have 

Cm(1s„(o'); 1mi,„(o') ® 1m„,i(o')' [C,a;,y]) 

^ r t([c, X, y])o^'A^vt(o'r 1mi„„(p) , [C+, ^]) [C, ^, y] ^ [C+, ^] e u+„(fc'); 
lo [C,x,2/]^[C-,z]eu-„(fc') 

where 

Cm(1s„(o');1mi,„(o') ® lM„,i(o')JC,a;,2/]) = J ls„(o')([C]h) (^^^-(h) (1mi,„(o') ® 1m„,i(o'))) (2;, 2/)?A(h)dh; 

H(fc') 

0^.,^(1u+(„,);1m,,„(o),[C+,^]) = J lu+toodC-'lh') (c.^_(h')lM,,„(o)) [z^WW. 

H+(fc') 

It is easy to see that [t^,x,y] O [C^i-s:] e Unm(^') "if md only i/ val (A[^ j, j,]) is even. 
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In particular, when n = m, the above orbital integrals become the following much simpler ones 
Cm(1s„(o'); 1mi,„(o') 1m„,i(o'): K,x,y]) = J ls„(o')(5~^C5)lMi,„(o')(2;ff)lM„,i(o')(5~V)'?(det5)d5; 

GL„(fc') 
U+(/c') 



Proposition 5.12. Let n — m or m ~ 0, ifval (A[^ j^j) is odd, then 

Cm(1s„(o'); 1mi,,.(o') ® 1m„,i(o')' [C,x,y]) = 0. 
Proof. It follows from the similar argument in the proof of Proposition 4.15 by noticing that ,,(0') i^) = 

1m„i(o')(*2;)- 

□ 

It seems that, when n > m, the above vanishing result is not that easy to proof. 



5.5 Proof of the fundamental lemma for U„ x U„ 

In this section, we prove the fundamental lemma for the case n = to in positive characteristics. The 
proof uses a similar idea of relating orbiting integrals to certain problems of counting lattices, following 
[Yun09], and then reduces to an identity which has already been proved by Yun. In other word, the 
fundamental lemmas for both the case n = m and n to + 1 (the Jacquet-Rallis case) will be implied 
by the same identity. 

Let k' be a p-adic local field with ring of integers 0', uniformizer zu, and q = |o'/n7o'|. Let k/k' be 
an unramified quadratic field extension with ring of integers and ^ j € with j'^ = — j. Let val be 
the valuation on fc^ normalized such that va^tu) = 1. For two full rank o-lattice Ai, A2 in some finite 
dimensional fc- vector space V, we define 

leng„(Ai : A2) := leng„(Ai/Ai n A2) - leng„(A2/Ai n A2) 

where for Ai D A2, 

|o/n70| (7^ 

We have also the same notion for 0' (but of course replacing q"^ by q). 

Let us first review some constructions in [Yun09, Section 2]. Let n > 1, for any pair {a,b) with 
a = {ai),b = (bi) G 0" and a„ G 0^ , we define an o-algebra 

Ra,o = o[t,t-']/{r - aii"^! + • • • + (-l)"a„). 

Hence Z'^ := Spec Ra,o is a subscheme of Spec x Gm which is finite flat over Spec of degree n. Let 
R^ j, = IIomo(Ra,o, 0) and define an element b G ^a.o from the datum b by the formula 

b : Ra,o (5.17) 
fK->6,, i = 0,...,n-l. (5.18) 

It induces an Ra^o-linear homomorphism 7^ ^ : Ra,o — > R-a t>y the pairing 

Ra,o €5 Ra,o > 

(u, v) t— s> h{uv). 
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Let be the involution on RcS(,/(,; (Spec o x Gm) which is the product of t on Spec o and the involution 
1 1— )■ on Gin. The subschcme invariant under i!) is the unitary group (scheme) Vi o/o' over Spec o'. 

Recall that S„(o') = {C G M„(o') | (C = In}- For [C,x,y] e S„(o') x Mi^„(o') x M„,i(o'), let a, = 
ttidC, X, y]) ~ Tr ^/\' C^^ and bi ~ bi{[C, x, y]) = x(^''y be the invariants. Then the corresponding subscheme 

is stable under 'd, hence determining a subscheme Za of Ui finite fiat of degree n over Spec o'. 
Let Ra be the coordinate ring of Za which is a finite flat o'-algebra of rank n such that (E)o' o = Ro,o- 
The Ra,o-liiiear map 7^ ^ descends to an Ra-linear map 7a, b : Ra Ra: where Ra = Hom(,'(Ra, 0'). 

Let us assume that j,] ^ 0, then the image of 7a, & is co-finite in R^. If we identify Ra as a 

submodule of Ra, then lengp/(R^ : Ra) = val [A[^^j.^y]). For each < « < val (A[^,2:,y]), let 

Mi,a,6 := {Ra-lattices A | Ra C A C Ra and leng„,(R^ : A) = i} . 

We remark that for the orbital integral C'^(ls„(o'); 1mi „(o') 1m„ 1(0')' [C, 2;, y]) to be nonzero, there 
must be an element locating in S„(o') x Mi_„(o') x M„,i(o') in the GL„(fc')-orbit of [(,x,y]. Then we 
have 

Proposition 5.13. Lei[C,x,j/] e S„(o') xMi,„(o') xM„,i(o') be regular and hence such that val (^A^t^ ^c yj) > 
0, then the orbital integral 

val(A[^_^_^]) 

Om(1s„(o'); 1m,„„(o') ® 1m„,,(o'), [C,^,2/]) = (-i)™^(^K--i) (-1)' l^*.-."! 

where .^.j,] is defined in (5.13). 

Proof. Let V = M„.i(o') be the o'-module and we identity with Mi,„(o') by matrix multiplication. 
Then V = V{k) := V k. RecaU that 




ls„(o')(5 ^C5)1mi.„(o')(2^5)1m„,i(o')(5 ^y)v{<iotg)dg 



GL„(fc') 



(5.19) 

and the measure is the one such that GL„(o') gets volume 1. We define 

Xlc.,y] ■■= {9 e GL„(fc')/GL„(o') I g-\g £ S„(o'),xg e V\g-'y G V,val(dct5) = i} ; 
Yjj. ^ ^j := {o'-lattice L C V(A:') | C oi, .t G L^, y G L,leng„,(L : V) = z} 

where = {v G V^(fc') | vv' G 0' for any v' G L}. There is a bijection Xj'^ ^ Y^^ ^ given by 

On the other hand, we define an fc-lincar map 

y' :Ra(fc) ^V(fc) 

^ Cy 

which is bijcctive since x, y] is regular. Here, Ra(fc) is the underlying fc-module of the fc-algebra 
Ra ®o' k. By the definition of 7^ j,, the following fc-linear map 

x' = lib o (y'")"' : Ra(fc) ^ Kik) ^""^ ' V-(fc) 

is given by f t-^ . It is clear that y' (resp. x') descends to a fc'-lincar map y : Ra(fc') — > V(fc') (resp. 
x:Ra(fc') V^(fc')). 

For any L G Yj^ ^ we claim that y^^(L) G M^^j^rp^_ ^ j^. In fact, y^^{L) is a lattice stable 

under Ra by the construction. Hence we only need to show that Ra C y~^{L) C Rq. Since y € L, 
y^^iy) — G y~"'^(-^), where 1r^ is the identity element of the algebra 1r^. Hence 

Ra = Ra • 1r„ c Ra • y"'(L) c y"'(L). 
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Similarly, since x G V', x-i(x) = 1r„ G x-i(L^) = (y-i(L)) . But (y-i(L)) is also stable under 
which implies that 

Ra - Ra • 1r„ c Ra • {y-^^Y G {y-^^Y 
i.e., y~'^{L) C R^. Moreover 

leng„,(Rr : y-\L)) = leng„,(R^ : y'^Y)) - leng„,(L : V) = val (T[^^,,,,) - i. 

Conversely, for any A G M^^j^-^^^ ^ ^^)-^.a.b^ ^'^^ ^ = ^hen R^.o • A = (Ra • A) ® o C A ® o, 

hence (^L G oL. The fact Ra G A implies that y = y(lR„) G L and the fact A C R^ implies that 
X = x(Ir^) G . The length part is clear. Hence, we prove the claim. 

Then 



(5.19) = (-1)' 



i=0 



X 



1=0 



™1('^[C,x,b]) 

4=0 



□ 

Now we consider the orbital integral on the unitary group. We fix an element /3+ G Her„(fc) nGL„(o) 
which defines a unitary group scheme U^^^^^,, whose generic fibre U^^y^, x fc' = U+ viewed as a subgroup 
of Res/;/fc/GL„_fc. Now we simply view U+ as defined over o'. For [C^,^] G U+(o') x Mi^„(o), we recall 
that z* = (/3^) ^ ^z'^ G M„.i(o) and the invariants (a, 6) G o^". Then the corresponding subscheme 
Z'^ = Spec Ra,o is also stable under i?, hence determining a subscheme Za of \ii,o/o'- Let R^ be the 
coordinate ring of Za- We have an R^ p-linear map 7^ : R^ R^ ^ identifying Ra as a submodule 

ofRXo- 

We define the following morphism between fc-modules 

z* :R,(fc) ^ V(fc) 

f ^ {C+Y z* 

which is an isomorphism if [C^,-z] is regular, which we will assume in the following discussion. The 
hermitian form on V(fc) defined by /3+ induces the following hermitian form on Ra(fc) through z*: 

(-", w)r = b [uv"^) 

where b : Ra(fc) — > fc is defined in (5.17). 

For an o-lattice A+ C Ra(fc), the dual lattice under (— , — )r is the o-lattice 

(A+)^:={i>GRa(fc)|(i,,A+)RCo}. 

We caU A+ self-dual if (A+)'' = A+. We define 

Nc^b := {self-dual i?Q^o-lattice A+ | Ra,(, G A+ G Ra,o} ■ 

We remark that for the orbital integral ©^'^^(ly+^j,,-); „(o)i [C'^^j-^:]) to be nonzero, there must be an 
element locating in U+(o') x Mi,„(o) in the U+(fc')-orbit of [C^,z]. Then we have 

Proposition 5.14. Let [C^tz] G U+(o') x Mi^„(o) he regular and hence such that val (Aj^^+^^j) > is 
even, then the orbital integral 

Oi,'A^Vt[o'V 1mi,„(o), [C+, ^]) = |Na,b| . 
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Proof. Recall that 



^^'./^(1u+(o');1mi,„(o),[C+,^]) = J lu+(o')(5'"'CV)lMi,„(o)(^5')d.g' 

U+(fc') 

and the measure is the one such that U^(o') gets volume 1. We define 

^[C+,.] - {5' e U^(A:')/U+(o') I g'-\+g' G Vtio'), g-'z* G V(o)} ; 
y[^+_^j = {self-dual o-latticc L+ C V(fc) | C^L+ C oL+,z* e L+} . 

There is a bijection -'^^[^+,2] ^[C+.^l given by g' n- p'V(o). We have another bijection 
given by 1-^ (z*) ^ (i^) whose proof is similar to that in Proposition 5.13. Hence we have 



(5.20) \X 



|N„ 



(5.20) 



□ 

Theorem 5.15. //char(A:) = p > max {n, 2}, t/ie fundamental lemma for U„ x U„ ('c/. Conjecture 5.11 ), 
i.e., the following identity 



mIJ1-S,.{o')' -^Mi,„(o') 
'(_^)val(T[,,.,„j)0^,_^(l 





.1(0')' [C>a;,2/]) 



U+(o')' -'^Mi,„(o)j 



[C+,z]) [C,x,2/]o[c+,z]eU+„(fc'); 
[C,x,2/] o [C-,z] e U-„(fc') 



/io/(is /or [C, a;, y] regular. 

Proof. The second case of the above identity has already been proved in Proposition 5.12. But Proposition 
5.13 leads to another proof by noticing that, first val (Aj^ ,j. ,^]) is odd and [Q, x, y] satisfies the assumption 
in that proposition (otherwise, the orbital integral will be automatically), and second A 1— >■ induces 



a bijection q j, 



M 



val( A[j- 2, y] ) — z,a,6' 



For the first part, we need to prove the identity 



val(A„,6) 
i=0 



(5.21) 



assuming that (resp. [C+,2]) satisfies the assumption in Proposition 5.13 (resp. 5.14) and 

they have the same invariants (a, fe). Here, Aa^b is the determinant of the map 7^ ^ under the basis 
{1, t, t""^} of Ra{k) and the dual basis of '[V^{k) and hence val(Aa,fc) ~ val (A^^ j,]) . We remark 
that the equality (5.21) is a property purely of the o'-algebra Rq and the Ra-linear map 7a, t : Ra ^ Rq ■ 
By the argument in [Yun09, Proposition 2.6.1], we can find ai,bi € fo' and an isomorphism of 
o'-algebras p : Ra Ra such that the following diagram is commutative: 



Rn 



Ra 



7a, 6 



where the algebra Ra and the map 7^^ are defined in [Yun09, Remark 2.2.5]. Since [C,a;, j/] is regular 
which means in particular that Rq (3 fc is an etale fc-algebra, hence Ra ^ k is also etale. Applying the 
fundamental result [Yun09, Corollary 2.7.2] (and also their notations) to Ra and 7-^, we have that 



val(As.s) 

E (-1) 



i=0 



Af' 



K 



loc 



for char(fc) = p> max{n, 2}, which implies (5.21) immediately. 



□ 
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Remark 5.16. If char(fc) = and p is sufficiently large with respect to n, the transfer principle in [Yun09, 
Appendix] should also apply to our case and hence imply the fundamental lemma in characteristic 0. 



6 Appendix: A brief summary on local Whittaker integrals 



In the appendix, we summarize some facts about certain integrals of local Whittaker functions which will 
be used in this paper. AU the resuhs arc contained in [JS81], [JPSS83], [JS90], [CPS04] and [J09]. 

Let fc be a local field, t/j : k ^ he a nontrivial character. We denote | | = | \k, M^.s = Mr.s{k). Let 
TT (resp. a) be an irreducible admissible representation of GL„ = GL„(fc) (resp. GL„i = GL„i(fc)). Let 
Wii}) = Indg^^„"(V') be the space of all smooth functions W{g) on GL„ satisfying W{ug) = ip{u)W{g) := 
V'('"i,2 + • ■ • + u„-i^n)W{g) for all u = (utj) £ Ui^, the group of upper-triangular matrices with all 1 
on the diagonal. It is a smooth representation of GL„ by right translation. Let V^^ be the space where 
TT realizes. If k is archimedean, then we take 14- as the canonical Casselman-Wallach completion of the 
corresponding Harish-Chandra module of tt. A fundamental theorem of Gelfand-Kazhdan and Shalika 
posits that there is at most one GL„-cquivariant map, up to a constant multiple, from V^r to W{iJj). If it 
exists, then we say tt is generic. Being generic is independent of ip we choose. Same arguments apply to 
(J. In what follows, we will assume that tt and a are generic. We denote by W{tt,iI}) (resp. W(cr, ■0)) the 
nontrivial image of (resp. Va-) in W{tp) (resp. W{ip)). Moreover, 



W e 



where w„ 

and Wn^rn 



Wn-l 
Irrj 



is the longest Weyl element of GL„. Moreover, we let = [0, 0, 1] G Mi.; 



w„ 



Let W e yV(7r, cr), W G W((t, tp) and $ G S{Mi^m), we consider the following kinds of integrals 
• For n > m and 0<r<n — m— 1, 



• For n > m and 1 < r < n — to, 

*^(s;M^,VF-;$) = 

• For n > m, 

4'o(s;W^, = j W 

(7im\GL™ 



.9 

X Ir 
In-m 



W-{g)\ dot gl'^-'^dxdg (6.1) 



W 



/ 


.9 











\ 




X 


Ir-l 










y 





1 







V 











In — m— r 





9 




W {g)^{y)\detg\' "^'"dydxdg 
(6.2) 

W-igMerng)\detgr'^dg (6.3) 



We denote by 

X^(7r X a) ={^'ris;W,W-) \ W £ W(7r,cr),l^- G W(cr,V^)} ; < r < ra - to - 1 
Z^(7r X cr) ={*^(s;W^,H/-;$) | W G W(7r, cr), W^" G W(cr,V^),$ G 5(Mi,„)} ; < r < 

which arc linear spaces over C. 
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Proposition 6.1 ([JPSS83], [JS90], [CPS04], [J09]). We only state the following results for k non- 
archimedean, while the statement for k archimedean can be found, for example, in [J09, Section 2]. Let 
TT and a be as above and Xa the central character of a, then: 

(1) Each element inXr{'K'xcr) and I^{Trx a) is absolutely convergent for ^{s) large and has a meromorphic 
continuation to the entire complex plane; 

(2) There exists a unique function L(s,Tr x a) of the form P(q^'')^^ where P G C[X] and q is the 
cardinality of the residue field of k, such that 

Ir{-n X a) ^ I^{7T X a) = L{s,Tr x a-)C[q~ " , q"] 

for any r; in particular, for any r and sq G C, there exist W , W~ and possibly $ such that ^'r(s; W, W~) / L{s, ttx 
(7%^,^ or-^r{s]W,W-;^)/L{s,T: X a)\s=so is in . 

(3) There is a factor e(s,7r x a,ip), only depending on tt, a and tp, of the form cq^^^ such that 

^(,_^^~^~) = X.(-l) X a,^)^^——^ 

for n > m and 

^(,_^^^~^~) =X.(-1) eis,nxa,i.) ^^^^^^^^ (6.4) 

for n>m, where <& is the tp-Fourier transform of^: 

J ^{x)i{j{x*y)dx 

Mi,„ 

with the self- dual measure dx. 

Proof. The proof of these statements can be found in the hterature mentioned above, except (6.4) when 
n > m. For completeness, we will give a proof of (6.4) when n > m below, following [JPSS83]. 
By the functional equation, we can assume that < r < n — m. We let 



Wi = 

m' 



u 

Ir 

1 

In— m — r— 1 



which is in W{t:,iP). Then *r(s; M^i, VF^) = 'i'r{s;W,W~;^). To prove (6.4), wc only need to prove 
that "ifn-m-i-ris; p{wn,7n)Wi,W^) — ^'„_„i_r (s; p('w„,m )M^, ; <f>) which is true by the Fourier inverse 
formula. 

□ 

When k is archimedean or the representations tt and a are unramified, then the representation nMa, 
hence its L-factor L{s, nM a) and e-factor e(s, tt Kl cr, ^Z^) arc defined using the Langlands parameter. We 
have the following. 

Proposition 6.2 ([JS81], [JS90], [CPS04], [J09]). (l)Ifk is archimedean, then L{s,TTXa) = L(s,7rKlcr) 
and t(s, TT X a,'ip) = e(s, tt Kl ct, iJj); 

(2) If k is non- archimedean with o its ring of integers, let tt (resp. a) be an unramified representation 
associated to a semisimple conjugacy class A^^ G GL„(C) (resp. G GL„i(C)J. Let Wo (resp. W~ ) be 
the unique GL„(o)- (resp. GL,„(o)-j /iied Whittaker functions such that W{ln) = 1 (resp. W~{lm) = 
and be the characteristic function o/ Mi.„i(o), then 

"^ris; Wo,W^) = *r(s; Wo,W-;<i>o) = dot (l - q-'A^ ® A^y^ = L{s, tt x a) = L{s, tt ^ a) 
for any possible r. 
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Proof. In (2), the proof for the integral 4'o(s; Wo, ) when n > m, and for the integral ^'o(s; Wo, Wo ; $o) 
when n = m can be found in [JS81]. The rest will follow easily as in Proposition 6.1. 

□ 

Corollary 6.3. (1) When n> m, the Whittaker integral (6.1) defines a nonzero element in Honi//(7r(8) 
a, v), hence an {r,n — m — l — r)-Bessel model for n, a generic, by choosing a suitable basis like in Section 
2.2. 

(2) When n > m and r > (resp. n > m and r ~ 0), the Whittaker integral (6.2) (resp. (6.3)j (with a 
replaced by a ® fJ'~^ ) defines a nonzero element in Honi^f (tt (g) ct ® i^, C), hence an (r, n — m — r)- (resp. 
(0,n — m)-) Fourier- J acobi model for n, a generic, by choosing a suitable basis like in Section 3.2. 
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